Browsing Category

IT is business

Digital transformation, IT is business, Practical examples

Using Machine Learning in real life part 1/2

March 30, 2018

Everybody has been talking about Machine Learning, and everybody wants to get benefits of Artificial Intelligence. It is a new thing IT managers grabbing that old problem from inside the old locker and thinking: “hey! Maybe new Watson can solve it for me!”. But every time I hear someone new asking about how to solve a problem with AI, the problem looks like something never seen before. Every day a new solution is researched to a new problem. If every day a new will comes up, how can we identify what are the borders for AI? Since AI stands for “Artificial Intelligence” what is the “intelligence” border? What can and what cannot be solved with what we have today?

How to identify how hard it will be to find a AI to your scenario

Machine Learning projects can be split into three groups:

  1. Using a ready-to-use open API (this first article focuses here)
    • What is it? This is the fastest approach. There are a lot of APIs ready to be accessed and to be added to your solution. There are a lot of benefits, and you just have to pay for that. There is a table and more details below;
    • How long will it take? You can get results from some tests within one day;
    • Some benefits:
      • a) They are ready to use! You just have to plug them to your app. Anyone can do that;
      • b) Their suppliers will keep training the model as you go! So, it won’t ever be outdated;
      • c) The competition between suppliers will grant you always well trained models and non-stop improvements and updates;
      • The items above will be very expensive to reach when outside this approach;
    • Some restrictions:
      • a) It doesn’t belong to you. It means you can’t change anything on how it works. It’s just you asking: “hey, please classify this image!”. Then the answer would be: “cool! Your image has a woman on it”. But you can’t ask back: “what’s the woman’s hair color?”;
    • Conclusions?
      • If the open APIs fit your needs, don’t ask again and start using them by now! Don’t worry about suppliers grabbing your information or whatever like that. Once you pay for the tool, you have a contract where they say they won’t use your data. It’s the same thing of cloud;
      • If it doesn’t fit exactly your needs, try to understand how important is to have that 1% more of trust over that AI judgement. If you really need something more precise, jump to “training a model”;
  1. Training a model
    • Mid-term approach. There are a lot of different models ready to be added to a project, configured, trained and then used. I will talk about this approach in the next article;
  2. Building a model
    • Keep in mind it won’t be an IT project for a while. You will have to have people from physics, mathematics and specialists on your business, and really good information to add to your project in the very beginning. Once they finish the model (it can take up to 2 years. Maybe more), then it will turn into a regular IT project starting from “training a model”. Both of this projects (building and training) will take for sure more than 2 years of research and testing. If it is a key process to your company, don’t waste one more second and start this project. The sooner you start, the sooner you will get the benefits;

 

Cool! What are the ready-to-use APIs?

My suggestions are inside this table below. But the options are not limited to it. You can find many others. They are maintained by the biggest cloud and AI players. It means you can trust it, and probably for what they focus, they are the best you will find.

Feature Google IBM Microsoft Amazon
Chatbot related DialogFlow Watson Assistant and Virtual Agent Bot Lex
Video Analysis Video Intelligence Intelligent Video Analytics Video Indexer Rekognition
Image Analysis Vision Visual Recognition Computer Vision API Rekognition
Speech to Text Speech Speech to Text Bing Speech Transcribe
Text to Speech Text to Speech Bing Speech Text to Speech
Natural Language Classifier Natural Language Natural Language ClassifierNatural Language UnderstandingPersonality Insights and Tone Analyzer Language Understanding Comprehend
Translation Translate Translator Translator Translate
Trends search and analysis Trends Discovery and IBM’s Discovery News
Find patterns over unstructured text Knowledge Studio
Content moderator* Anomaly Detection
Jobs discovery Job discovery**

* Google, IBM and Amazon have content moderator built-in their products. Microsoft has this specific product looking for anomalies only.

** Job Discovery is a private tool available for only few partners.

 

Two examples to talk about the borders again
Image recognition

Just like the example above: a customer came to me asking about a solution to identify people on images. Great! Let’s use Google’s Vision! Vision identifies people on photos and gives a lot more information about the colors on that image, about places that image may contain, and etc. But then the customer asked me: I want to recognize if it is a woman. I said ok! And then: I want to recognize the woman’s hair color. Ok, all open APIs are off the game. Let’s find a model, train it and then get hair colors. For you to be able to answer those questions there is no shortcut. You will have to read the documentation of each open API you find and run tests on it.

Language defect recognition

Another customer came to me asking if they could give a microphone to their employees in order they could operate a system just giving voice commands. Ok, it’s not new. We could use a mix of speech-to-text and natural language processing APIs, let’s move ahead! But then the customer said the system should recognize internal terms like acronyms and words they invented to communicate with each other. Erm… it’s not possible. You can’t train ready-to-use APIs to understand your very own specific terms. The easiest way there was to suggest the operators to change the words for some others the system would recognize. Otherwise they would have to grab models, configure and train them to understand the new words.

 

Then, why don’t you give your first AI step over ready-to-use APIs evaluation? The sooner you start, the sooner you will understand how to approach that old problem.

Digital transformation, IT is business, Practical examples

Cloud is the new black

March 15, 2018

I heard “cloud is the new black” during a training session inside a Google’s office. You can guess it means that cloud is basic. But why did they say it? Google didn’t forge this beautiful sentence. Gartner did.

What Gartner means by “cloud is the new black”? In short terms, Gartner said the cloud Market is a US$ 1 trillion Market. Right now it’s just US$ 56 billion.

Google repeats that for comercial purposes of course. The reasons are the same Microsoft, AWS and any other cloud provider do: scalability, stability, abstraction of infrastructure processes, and the same bla bla bla. And I totally agree with their reasons. But without considering this technical questions, to the final business results $$$, why is the cloud so basic?

 

How does that come to our business?

Yes, there are already a lot of companies moving to cloud and starting their applications inside the cloud. But I still can easily find many companies still not even considering the cloud move. Inside my reality it’s hard to understand. How can they not see cloud benefits? How can they still use their machines and spend millions of dollars buying more and more storage every 6 months? For me it’s waste of time. I’ll explain why.

Physical space

The rooms where the machines are hosted. They cost Money. For few companies, I’ve already seen very expensive entire blocks inside noble areas like São Paulo being used to host… machines. They don’t need the datacenter to be that close to the offices. The latency doesn’t matter that much. I’m 100% sure. For sure if they remove everything there, and rent the space, the renting revenues will pay for a big slice of cloud cost monthly. What if they sell it? It would mean investments for areas that are needing that Money to innovate and be in front of their competitors. Because of that lack of money, their areas are wasting time. It’s waste of time.

Rework rate

Recently a datacenter, close to the company where I work had a fire issue. Many governmental and private companies, core and non-core systems, were affected. And where were the backups? Inside the same building. Because of the fire, the fireman and police didn’t allow the technical team to get there and move the information to another datacenter. The replication wasn’t automatized. It caused more than 12h of unavailable systems. Can you imagine any company inside any industry without receiving transactions during 12h? Imagine the financial area. Hard. Now imagine a factory without systems for 12h. They won’t sell for a whole day? You could answer “Yeah, but we can ‘take notes’”. The employees don’t remember how to hold a pen this far. They also won’t know how much they produced of what they produce. They won’t know how much they spent producing things. But the main thing here is the overhead that will be created inside those companies to put everything back to systems. Talking about Brazil they can even get a ticket from government because of some missing transactions on that day. What all that means? Waste of time.

Why not cloud?

I have a client who runs a retail solution on cloud. The solution has been running for the last 2 years without downtime. Is it a core solution to their business? No, it’s not. But the fact that they don’t have headaches with that small system saves them time to think on other things. Saves tickets on traditional infrastructure teams. Saves their mental health also. Of course the cloud itself is not the only answer for the application stability. They do care about quality on their development process. Then all these benefits come easily.

 

When will companies decide to migrate to cloud?

It all makes me think that using cloud is related to maturity. Now a quick link with internet-related startups: companies who grow unbelievable percentages every year in the entire world. The biggest part of them has the cloud in common. Most of their business models wouldn’t be possible without the cloud.

The traditional companies, which already felt startups “bothering” their market shares, are moving, or already moved to cloud.

Why does that happen? Because the cloud gives them the speed they need. Things I’ve already seen in on-premise VS cloud environments:

  • A new environment to create a new app can take up to 1 month to be released by the infrastructure team to the development team to start work. It’s one month less on that project. Within cloud it’s solved in less than an hour.
  • Analytics information being generated only with the data considering the day before the current. In cloud you can have live information to take your decisions.
  • Analyzing petabytes of data without having to do that on the weekend, when there’s no concurrency with other systems running. In cloud you can do that whenever you want without buying millions of dollars of infrastructure in advance.

All these examples want to say the same thing: when the companies start feeling they are being left behind because they are slower than their competitors (either it is a startup or not), they will change.

 

So, why cloud is the new black?

So… Getting back, why cloud is the new black? Because it means saving time. Because if this text made you remember of any issue you are having, or you may have inside your company, it means you will run after it to solve. It won’t be a short run to find all the responsible for everything and asking them to change to your new conceptions. It will take weeks. Months at least. Those weeks or months spent by the team looking to fix or prevent something to happen, means weeks or months not looking to improve the business, not looking to be in front of competitors. The IT area is not the support anymore. It can’t JUST be prepared to whatever the other areas will demand. It HAS to be the one of the leading business areas. And why that is true? The IT guys know what technology can do. The other areas don’t.

Digital transformation, IT is business

It’s time to take a look at Machine Learning

February 28, 2018

Machine Learning (ML) is the word for the moment. Just like SOA has already been, just like DevOps has already been. Now everybody talk about ML. Everybody want to get its miraculous benefits but just few of them all really know what it is, its possibilities, and the shortest path to start getting its results and leveraging with it.

What is ML?

Wikipedia says “Machine learning is a field of computer science that gives computer systems the ability to “learn””.

Ok, got that. But what is Machine Learning? What does it can achieve? Let’s get to some examples:

Netflix recommendations

Netflix told, during Cassandra summit on 2016, that everything it shows are recommendations. The slides are available here (there’s also a Youtube video at the end). And people like it! 80% of what we watch are recommended. We don’t actively look for it. It’s one click far.

They got 5.2 million new users from may to june in 2017. If you were on of them, you didn’t fill a long question set giving them information about which genders you like the most, not even what you don’t. Also, they don’t have the largest number of interns in the world to read 5.2 new user’s data and humanly suggest movies for them to watch. Those recommendations were made and keep always being made and updated within few minutes after you show any kind of sign you like something. Watched something? Flagged as “watch later”? Flagged as “like” or “dislike”? It all counts. It all updates your profile and new suggestions may come.

Google Maps

Now take a look at this second example. This Google’s Street View image shows a parking lot in Primeiro de Março Street, at São Leopoldo (south Brazil small city):

I’m 100% sure the parking lot named “estacionamento beto” didn’t send their data to Google Maps team. They didn’t fill a form telling they are at São Leopoldo. Their owner probably have a mobile phone with internet access. But probably they didn’t even know it is possible, and maybe they didn’t even know what is Google Maps. But they are there:

The Machine Learning here is something already touchable using a Google API to analyze images. They analyzed the Street View image, and the machine knew it was something with commercial purposes. Then the algorithm automatically “registered” it as something to be shown in the map. If you look at it at Street View, you will see that this street block is almost entirely residential. Google’s algorithms knew what’s the different between residential buildings and commercial buildings.

Few others
  • Facebook suggests you to tag friends at their faces when you upload a new photo. It’s Machine Learning. Facebook doesn’t have one picture of each people in world to compare when they receive new pictures;
  • Almost all e-commerces we know nowadays do that, but Amazon started: products suggestion based on the last purchases you done;
  • Getting information from your bank account straight on Facebook’s chat. It can recognize many different questions for “whats my balance?”

 

How to benefit?

It’s very easy to think on new scenarios, more or less complex. Automatizing predictions or anything that today requires a human mind, is, or will be, possible.

Example! We know our stock market has patterns on its behavior and it’s affected by many things. Today many people spend their whole day (or life) analyzing data and giving predictions to others about what to do with their stocks (buy/sell orders).

The red dots (mine) on data above (from tradingeconomis.com) show moments when the market started cycles of loss over the last 10 years. During all of these examples we already had a set of technical analysis that helped us to identify when it’s more or less likely to happen again. But it (still) takes a lot of people distributed across many companies to MAYBE find something significant out. Having a good Machine Learning model can change this entire market. It can change into something unfair to put money, since parts of the analysis MAY become predictable.

 

But how to start?

A good new Machine Learning model invariably goes through a huge data quantity. The data is required to train the model and then achieve the real business goal. A good start is to set up a structured base of data to be used. You will also need a data scientist. Don’t worry, this scientist is not an alien. They are potentially very easy to find. But it’s subject for a new post.

Avoiding problems, IT is business, Practical examples

5 big companies mistakes moving to cloud without a good plan

February 21, 2018

A lot was discussed, and still have been, about cloud journey. The worldwide big companies already have their strategy turned to cloud ever since the applications start being planned. Potentials of security, scalability, autonomy and many other related subjects are not approached on time of project discussion anymore. The cloud doesn’t generate doubts or lack of trust anymore. Once going to cloud is an old subject, beside the attention points already brought, I discuss here some errors I’ve already seen, as a way to support and contribute to new moves:

 

Face cloud as a side-need

The cloud is the main actor on applications. It’s very common to see companies starting their journey with simple backup or storage routines on cloud. This way they can start dismissing their own datacenters. It’s an important step, many times needed to show safety to a skeptical to changes board. But rollback, backup, disaster recovery, and many other routines, which used to be hard to implement to OPS teams, are now trivial to big cloud players. Using cloud having only this objective is to underestimate cloud’s potential.

Self-managed services have a very high level of automation on the points shown above. Get the benefits from processes automation, and don’t waste time and money researching, planning and implementing. The not self-managed services also have a lot of automation built-in. Backup, rollback and DR are now the basics.

 

Decision matrix negligence

The option for one or another cloud provider is something very important and for many times underestimated. Selecting a cloud provider is like a marriage and can bring troubled relations! I’ve seen companies trying to run away from its 10 or 20 years traumatic contracts with giants like Oracle or IBM, neglecting cloud decision matrix. They ended up signing new contracts leading to new 10 or 20 years contracts with provider X or Y.

Big players like Google, SAP, IBM and SAP allow a very high level of customization. Keep it in your mind during the decision. Those will allow to configure everything needed to a big application. They cover dependencies, integrations and relationships with other systems, specific engineering practices and etc. It’s also very common that complex applications can aggregate benefits from multi-cloud operations. As an instance: infrastructures that has lower prices on provider X can get benefits from FaaS Bigdata services from provider Y, which is faster treating large data volumes. This way one single application can be distributed over more than one cloud provider and also over more than one geographic region. It allows business objectives like cost saving, delay reduction and access to specific features from niche players.

Beside the big players, there are many niche providers of PaaS and FaaS. Digital Ocean, Heroku, Umbler and Elastichosts as instances, are useful for not so robust applications. Those platforms have high abstraction levels, and it cuts the development/operations team learning path.

 

Low knowledge of the potential of the chosen cloud

The savings that can be reached using cloud resources, like rollback, DR, backup (mentioned above), monitoring, alerts and automated actions, are very high. Consider it when executing your migration project!

Evaluate PaaS services to run applications or FaaS for network tasks, Machine Learning, deploy, tests and other. As an example, I’ve seen that every company that had software running over on premise infrastructure, ended up at sometime developing something to improve its deploy and testing tasks. It means that, if 10 companies developed similar software, costing 500h each, they would have spent 5000h creating almost equal software. Having services like CodeDeploy, Device Farm, Fastlane and Firebase, this kind of development is not needed anymore. Fast access to these services saves many development hours. It gives more speed to companies and it means they will be able to answer faster to their customer needs.

 

Low tracking on investment

Do not treat cloud budget as a black box. Track the investment via capable people and separate it by application/business/whatever making sense on that reality. This way true decisions will be taken, contracting or refusing some evaluated service. This is one more practice to prevent the “marriage” with a cloud provider get similar to those contracts of many years with Oracle, IBM and etc, which all big company try to run away nowadays. We have already made mistakes in the past. Let’s learn with them!

 

No updates

As an example, it’s still common for companies to invest amounts of money buying different mobile devices to test apps. Lack of knowledge about services that allow to automatize testing on those devices make that money to keep being wasted and the benefits of automation not reaching the development process.

With more and more niche competitors growing, the lack of updates to the responsible team can be a big problem. Not knowing good alternatives to current services forbids you to discover and use new benefits.

Customers, Digital transformation, IT is business, Practical examples

Software buying process on IT evolution

February 14, 2018

The more different views you have about the same subject, the more information you will have to take your conclusions and make decisions if needed. This first article about IT evolution had a sight from developers and a very superficial business view. This new article aims to look from the perspective of clients buying software and how they are different even inside the same industries.

 

How software used to be bought

In past years, companies used to buy software like any other commodity: I want 5 cars with 4 wheels each, manual transmission and white painted. They didn’t mind on how their software is produced, or any kind of good practices to apply on the process. Yes, inside the software requirement documents were sections regarding security, protocols and things like that, but at the end of everything, the lowest price would win.

 

Old (not too much) decision matrix

The decision matrix (example below using a car buying process) were very simple when selecting the criteria. They were also superficial. The high level criteria were easily like: security levels, performance, proposed schedule, price, scope coverage, success cases, knowledge on selected technology, etc. With this decision matrix, the weights are given according to each project. But the price and schedule, after all the criteria evaluation would still determine who wins.

On the example above: what exactly is comfort and styling for this buyer? Safety can be evaluated using governamental public data. This is a parallel to show how none of the criteria is detailed.

Anyone saying that could cover the entire scope, under the asked schedule having a fixed amount of money could win the fight for the projects. If they didn’t have knowledge enough, it wouldn’t make huge difference. After everything, software was treated like any other product. It doesn’t matter how it’s made. It just matter that it works.

 

Recent decision matrix

Leaving behind government industry, and few others not affected even a bit by digital transformation, the reality has been changing.

What I’ve seen lately is an increasing spend of time on detailing the criteria. For each of the criteria, the buyers want to know what the supplier have already done, how they plan to conduct the solution and also want to evaluate alternatives for everything.

  • Security: ensure the information exchange will occur under a controlled environment? Tactics for code obfuscation. Platform natural security concerns. Cloud provider certifications, etc;
  • Schedule: from “a detailed schedule” (which is never achieved) to “a macro view of time and a mix of agile methodologies to be followed”;
  • Success cases: show where/when similar solutions were already created;
  • UX:  from “the system has to have good usability” (completely subjective) to “it must follow UX guidelines from Google and be conducted by a specific professional, not by developers”;
  • Performance: from “the system has to load under 10 seconds” to “each endpoint must answer under 2 seconds”;
  • Scope: from “do everything desired” to “let’s do the most we can inside the schedule”;

The companies want to know how their project will be conducted in details, and they also want to put their opinion on it. Since the digital transformation is making companies to turn their core into IT, the IT is not a support anymore. So now they are able to suggest things and to talk at the same level of the consultancies.

With this comparision and evolution scene, the MVP mindset is very clear. Also the goals to achieve faster time-to-market and faster revenues.

 

Different requests inside the same industry

The report above is true for 100% of companies on the most mature industries in Brazil. But for the rest of industries, using retail as an example, it’s not:

There are stores who are already evolving their ecommerces for security, performance, scalability, stability and the most important: user experience. But there’s also those who treat their online stores as something needed. Is common to copy layout practices from concurrents. Also, security is expensive. Then some companies have a budget to lose due to hacker invasions, instead of investing to security.

The good news is that the market is evolving faster and faster. Soon no industry will be behind.

Entrepeneurship, IT is business

Business automation benefits and impacts

January 22, 2018

We automatize things since the very beginning of human history. The windmills powered by the water coming down the rivers can be considered as an automation to process seeds. A lot of discussions have been growing lately due to the automation factor. Mark Zuckerberg and Bill Gates have already pronounced themselves about the impacts of the recent automation initiatives.

Talking about recent IT projects, a comment I’m very used to hear from many groups of people is the question about what will happen with people’s current labor.

 

The automation always existed

As mentioned on the article “A vision about IT evolution“, the IT was always faced as an arm for automation. Since it was created until today, we are used to automatize sheets so the work can be done in a decentralized way. Many people working on the same activity creates the need for a system. When its built they can communicate and get advantages of each other information. E.g.: The guys from logistics have to warn people at the gates to allow the trucks to go out when they are done loading. If there wasn’t a system to help, someone calling other person every 5 minutes would be needed, or even someone running through the whole plant during the day.

But the industry focus was kept inside the industry for many years, since the companies were who had machines and money to buy them.

 

Now it changed its scale

Nowadays everyone has a smartphone, and internet. With that, many new business were created. Now the automation, and new businesses, can reach millions of people easily.

  • Uber came and created a business model that connects customers to suppliers. Uber, iFood, Archie and many others get benefits from that. There is no intermediate involved anymore.
  • Digital banks – How people face money today? The agencies can be found empty easily nowadays due to the move of convenience at people’s hands. Big banks from Brazil don’t have ATMs anymore.
  • Retail – people get bored when they go to malls or stores because of many things. There is too many people competing for attention, the lines are stressing and there is always the risk on moving from their places to the stores. We can’t neglect that the biggest store of every retail brand is the online one.

 

New opportunities

As stated by the speeches of Gates and Zuckerberg, many people may face a change of their jobs. They can become obsolete due to this market change. But whenever a change come, many opportunities are brought with it. The responsibility of big companies and entrepreneurs, whenever they automatize something, is to let people do what machines can’t: think. Automation will bring several benefits:

  • Quality – every process will be ran in the same way. Then improving that single point will improve all of the processes executions;
  • Predictability – with maturity you will know what to expect from your tools. Then you can plan in advance;
  • Time savings – standardization brings speed naturally. Lets improve the single point to take the best allowed performance;
  • Metrics – using practices of telemetry, you can get metrics to take decisions about which point you should improve first;
  • Reduced costs – having all the benefits above, the direct ROI will be found easily;

For the companies and entrepreneurs, automation means saving money in a process. The people who will have their time saved already have a lot of knowledge on the business and on the process. Their new responsibility is to research and think on ways to improve the processes. Machines still can’t think like people.

 

What else?

During recent years, we faced a new mindset talking about services providers getting ease to find their customers. Uber, iFood, etc are examples of a business model created by one of the steps of automation history. When we talk about large scale automation, one of the steps to come has the blockchain influence. What it does is to remove the intermediates of handling money process, making one edge getting to another without assistance, and in this way money is saved. It also has potential to affect things not only related to money. What can be the next industries affected by automation at this scale?

Digital transformation, IT is business, Office

A vision about IT evolution

January 7, 2018

When the IT and the modernization it brought first started arriving the industries, there on the 1960’s, it was faced as something not wanted, but needed. The IT had its own language and terms, like the other areas, but for some reason it was not serious as the other areas. The IT people used to be treated as the people who could make magic and make people’s work easier. But what we have seen lately is a huge evolution and change on how IT is faced.

 

First scenarios

I heard during my graduation, from a professor, the conclusion that IT is a support area only. And I agree it used to be treated like that. The IT team were the strange guys, their workplaces were the worst, even not having windows to look outside. And it can be the reality until today for some companies. But it’s changing fast.

Old IT department

As a software service company employee, for years I used to be called to talk to our clients about new projects. The final conclusion is that all of those meetings were the same. I was called to create something new that could make activities easier and faster to be done by someone. And after that a waterfall move used to happen: less time was needed, then less people were needed, then less money were spent to keep that process/department.

As an instance, for those projects the procedure inside the demanding companies were the same:

  • Identify the need: we are slow at creating contracts for our clients;
  • Start the strategy move to change: let’s automatize the contract parts, that can be automatized, and add collaboration so our writers can work faster and together;
  • Call the IT department: hey, we want a tool!
  • Reach the ROI: we will save US$ 50.000,00 a year having less people writing, and reaching less our law guys;
  • Then the IT department calls the consultancy company (me): please build this tool.

After the tool was delivered, the contracts area got faster, the company saved money, and the whole IT talking was finished. Talking business, our clients were demanding about improving and automatizing parts of processes. The IT used to be faced as the support area which could solve problems or improve processes of the products areas.

 

Then the digital transformation

 

The customer experience focus, is pushing one of the most mature steps of digital transformation we have this far. The IT becoming the companies’ core. This move has been happening based on a world connected to the internet. Since everybody has a smartphone, and everybody want to solve things faster, leaving behind unnecessary stuff (like lines, support waiting times, etc), the IT is the way to reach those people. The customer needs are changing faster and the time spent on that formal process can’t be handled anymore. It must be possible to make changes and adapt at the same speed of the market. Because of that, the IT must be available 100% of the time.

Using the same instance, but now changing to mature business, the new procedure to improve something would be:

  • Identify the need: we must create a new feature to our final customer;
  • Start the strategy move to change: let’s include the feature inside the app;
  • Reach the ROI: we will make US$ 50.000,00 a year with that new feature;
  • Then let’s do it; No more boring meetings are needed;

With this core move, the IT is getting more importance on the business scenario. It’s not the ugly son anymore. The financial industry in Brazil is one of the most mature industries in world, and is pushing the others. I have already seen arguments between the final product areas, which used to rule the scenario, and the IT. It doesn’t make sense anymore to keep these two areas separated.

 

The next industries

This move seen in the financial area, which goes through every single company, isn’t common to all industries yet. For the next industries affected, its easy to understand the more B2C the company is, the faster will be its rush to adapt. Retail is a good candidate to be the next one. But what about the rest? When the factories will get to it?

Avoiding problems, Digital transformation, IT is business, Practical examples

What is telemetry, why it’s important and how to start!

December 17, 2017

The application stability has been a more frequent concern for companies specially when we talk about high value applications. Every time a core application stops working, many money is lost or many money stop being made. Because of that, a lot have been said about telemetry for applications more and more often. But what is telemetry for software actually and how to get benefits from this practice?

 

What is telemetry?

Telemetry is the act of measuring something remotely, by distance, and automatically.

Talking about software architecture, the telemetry is already very easy to find. Some simple examples are the Chrome platform, the Windows, OSX, Android, iOS and Sony’s Playstation OS operational systems, and also your mobile and desktop apps, such as Spotify and Microsoft Office. What these softwares do is to operate and gather all data that matters to its work. Then they send this data, naturally just if you allow, to their manufacturers (Google, Microsoft, Apple, etc). The next step, when they are grouped, is to analyze the data and change things if they have to. The core intention is to improve the systems so they can operate in many different environments having its proper behavior.

So the main thing about telemetry is to operate, gather data, analyze, and then improve the system code to reach a better behavior.

 

Why telemetry?

The telemetry can bring a lot of value to the business. Lets explore an example. Imagine you have an app running, which has a non-interactive FAQ screen to your users. Once your users get there, they will stop using your call center service because they have already found what they were looking for. This means money saving to your company. Now imagine that one of the answers (a quick how-to video) of this screen, for some reason, stops being shown (stops working properly). If you don’t have something checking for this screen’s healthy, it will be hard for you to notice, because we don’t have people browsing on systems 24/7. And then you will depend on some user good will to TELL you that the screen stopped working. It will happen sometime, but before that, your call center will start being called more and more. That’s waste of money because of software bad behavior.

The telemetry can be used to check many levels of service operation:

  • Very deep information, like machine’s CPU and memory. Are they green?
  • Do the number of machines up match your historic knowledge about how many were supposed to be up to support 1, 2 or 3 thousand of users at the same time?
  • Are the core webs-services your customers access every time up?
  • Is the final screen the user uses to login up?

Then when a telemetry practice is watching the important things on your application, you will be able to take actions and prevent problems from happening.

 

How to start?

A good telemetry implementation will depend on the size of information you will want to store and to analyze. The more information you have, the more infrastructure and knowledge you will have to have to process it all. It can even mean using bidata tools. But let’s talk about a new simple example, such as a system that receives vehicles security data from a company that sells insurance letters. A good way to start can be as following:

  • Identify why to measure: let’s measure because it is a core service that tells our customers how their loads are being transported over the roads;
  • Be sure of the goals related: the main goal is to keep the entire system running without down time, because every time this application stops, the contract allows the customers not to pay for that down time;
  • Identify what to measure: let’s check if all of our many data inputs are up. Let’s also check if the inputs are sending the same amount of data they are used to;
  • Set a strategy for measuring: let’s reach every end-point of the many data inputs. If they are available means its healthy. If they are not, its red. Then let’s read the amount of data received in the last minute. If it’s around the known number, its healthy;
  • Set an analysis strategy: can we automatize anything? If one of the endpoints is down, is it useful to restart the operational system, container or application server of the load balancer or the micro services? Or will it have to be shown in a dashboard to a human to analyze?
  • Implement ways to gather the data: let’s create the code to gather data and take actions, or to show it;
  • Show it! Now it’s time to show it. Is it useful to create a chart? Let’s show it using colors, so people can easily identify when there is a problem. If we need results fast, a good very first small step MVP could be opening a ticket on the infrastructure team;
  • Analyze: this is the most important time. It’s time to be critical and identify the root reason of the problem. Do not focus on the problem, but why it is happening. Why it is happening? Do we have problems with the parts that send data to us? Is the problem on our side? If yes, is our application running the way it should? Do we have to change something in our development process?
  • Take actions: through code or not, solve the things you found in all the steps above;

 

The telemetry can bring a lot of value to the business. It will give you intelligence to act before things happen. If your business is critical, it can mean a lot of money easily. It’s a common practice to many things in our administrative areas, like the PDCA mindset thought, why not do that for our software?

IT is business

3 reasons to speed up your development process

November 12, 2017

When a company needs a software to operate some process, the usual first step is to look at the market for something ready to be used.

When this software regards a vital core process, the decision of developing something new has to be taken. Whenever the decision of making good quality software is taken, it will probably take considerable time and will be expensive. The companies already know that. What if we could make the process of developing software shorter than usual? Why to invest on delivering software faster?

 

Work efficiency

The majority of companies of all segments have a backlog of internal desired software, and they are commonly generated by the IT area itself but also by the business areas. Since the business areas are requiring more and more software, we can see a move of the automation mindset coming up. The business areas have been understanding that if they manage to spend less time in routine tasks, which can be done by anyone, they will have more time to let people actually think on their jobs.

When people spend less time doing routine tasks, they have more time to criticize the process itself. This kind of will is amazing, because then the companies have the people who know best the process, actually thinking to improve it. Once a process is improved, less money is spent, or more money is made. Then we are actually improving the business.

IDC states that companies which managed to implement OpenShift, and used that to deliver software, had a 531% of average ROI in a 5 years time window. Also other technologies, such as Google G Suite, can speed up the development of software to meet simple workloads. So why not use fast development platforms and practices to speed up this backlog need?

 

Go-to-market speed up

Months being used to reach some point in that software is not an acceptable strategy anymore. The more time you spend developing something, the more likely that something will be to not fit the market. And right now that is the main concern of every CIO of companies who sell their products straight to final consumers.

Since the market is being flooded with a lot of new solutions, the software development speed must be increased. How can a startup put out a new reliable idea after a weekend? And how can it and put it to work after one or two months? If they can do that having no money, the big companies should do that easily right? This is the way the big companies will avoid to be replaced. IDC also states that the development life cycle can be accelerated by 66% using OpenShift.

 

Technological independence

Once you can find a platform or well defined process to always develop on top of it, a pattern will be created. The more they are used, the more familiar will be to your development team to use it. Beyond the spared time to set up a new project, the whole IT cost will be reduced:

  • The IT team productivity is improved: once there is a pattern to develop, it’s easier to run a new project, to train people and also to the OPS team to support that;
  • Infrastructure cost is reduced: when the team has knowledge about containers and microservices, the decision about which cloud to use can be changed easily. Then it’s faster to change to a new cloud provider with best cost benefit trade-off;
Avoiding problems, IT is business

7 attention points for cloud journey

November 4, 2017

The cloud journey, which is a subject still very common on IT areas of all companies that grew with on-premise systems, may have different complexity levels. To recent companies, with a small number of applications, the projects are commonly simple. But for companies with a long history of existence, it is usual for the project to take more than one year, just to migrate the applications and routines. When we start talking about updating the apps to use the advantages of being on cloud, this date gets longer. For both scenarios, many aspects must be analyzed, observed and monitored to guarantee the success. Below I list 7 important points to help on planning these projects:

 

Security: set politics and protect applications

As on the majority of system attacks, the social engineering applied to cloud context is the biggest concern for the majority of professionals. Weak passwords and lack of attention when defining politics of which user will have access to which type of actions on the environment are the main problems. Then, create well-defined politics and responsibilities for every single user that will have access to the environment. Each user must have permission to do JUST what is expected from him. The superadmin users are for one or two people on the whole company. Only they will have trust and maturity to orchestrate the whole team.

Another important thing of cloud security is the definition, with the development and architects team, of the company’s APIs access politics. One API which has capacity to make changes to sensitive data must have access, protocols and security designed and restricted to its purpose only.

 

Validation to avoid data loss

For data migration projects, such as databases, or files of any kind, validations must be done to ensure data integrity. This kind of move may have the intention of keeping backup, high availability, improve performance, etc.

For large volume migrations, such as TB of data, it’s important to check if the file number in the origin matches the destination. Another check is total size (counting bytes here): is it identical on both ends, for each one of the files? For databases, the same verification is useful at first sight, but since the database will probably be changed to work properly at the new environment (either changing to a self-managed service or not), the application integrity and its work validation are imperative, and must be done by people with deep knowledge on its behavior.

 

Legal: check if it allows the business

Talking about Brazil specifically, the legal part has been breaking governamental migrations for years. It also affects directly financial companies, because it forces them to keep their data physically stored at brazilian territory. Even with Brazil datacenters, the migrations sometimes are still not possible, because the cloud providers keep the automatic data replication, and some of them cannot guarantee that they will be located only at the selected region.

For other applications, not related to government and financial sector, the group of laws “marco civil da internet” allow the majority of companies to be free of those restrictions, and is a good starting point for consulting.

 

Architecture

Transfer applications from on-premise scenario to cloud on AS-IS format is an alternative when business requires agressive deadlines. But just changing the location of the server won’t solve problems related to the application architecture, such as performance, scalability, stability, telemetry, etc. When the cloud journey start to look at the way applications were developed, a lot of new benefits can be explored:

 

Architecture: coupling/lock-in – use self-managed services or not?

A very helpful alternative is the use of self-managed services. Services like AWS Lambda, Amazon RDS, Google App Engine, Google Cloud Spanner, and others, bring at least two strong benefits:

  • More speed to the development team.

Using the Amazon RDS as example, the time invested on development to connect an application to this service, instead of a regular native installed database is reduced drastically. It’s allowed by the native features this service has. Examples: automatic backup, fast restoring of the service in case of failure, automatic replication, etc. This benefits don’t have to be developed because they are available within the service. And it can be done through simple configurations.

  • Costs saving

This services’ abstraction, about how they operate, allow the costs with operation and configuration to be reduced significantly. The execution strategy for these services will be the cloud provider’s responsibility. It will use the resources the best possible way, because it has knowledge about the entire system.

This and other points must be evaluated considering the important trade-off of coupling and lock-in. The more self-managed services are used, the more dependency that application will have. Also bigger will be the lock-in with that cloud provider. The company who runs the application must have this scenarios mapped to know that, when a requisition of moving the application to a new place come (eg. another cloud provider), the time invested will be bigger, because an AS-IS move won’t be possible.

Architecture: costs – understand the application and find the best service to that business requirement

Understanding the application work and setting the best service to be used to solve a kind of problem is a way to reduce costs significantly. One easy example is the use of hardware by demand using automation. They are available at administrator console level and can be also triggered by APIs, that allow code to handle hardware, to reduce costs of that application.

As shown on the image above, the majority of applications has consumption peaks. The architecture and application planning will allow the elasticity of hardware usage to provide the features according to the time needed.

Another useful example is the usage of more than one cloud to the same application or service. We can find benefits by provisioning hardware at AWS, through its available services that reduce development time, and make them to consume Machine Learning APIs available at Google to handle images, as instance. This concept of hibrid cloud must always be considered aiming to find the best scenario of trade-off between cost and business needs.

 

Architecture: telemetry – having all registered and automatized to understand “where” is every application and keep tracking

When the applications start growing, and the hardware and resources to be operated increase, telemetry practices will be mandatory. An example: an application who runs 10 microservices, and each one of them is distributed on 5 machines. When a problem happens, its impossible to access each one of the 50 machines looking for where the failure happened. To solve that problem, some proactive and reactive actions can be taken. Below are some examples:

  • Reactive: programmatically make the application to detect the problem and create a ticket for the DBA team to analyze;
  • Proactive: programmatically make the application to detect one software failure and take actions to solve it. Example: check the service health that are inside one bigger, and detail it until finding that the problem root is a database who stopped responding. Restarting this database and creating a register to a deeper analysis by a human, can be something automatic.
  • Reactive: register, in one single location, all the application logs. This way, when it’s needed to add a software engineer to investigate the problem, he will access one single place, and not those 50 instances.
  • Reactive: identify that the hardware usage of one specific service is beyond its regular average and warn about what can be a hacker attack.

 

Architeture: latency

 

Whenever a cloud move is planned or is on going, the latency subject is at the table. It will always exist. But its necessary to check the business adherence and ways to avoid the latency to affect the system:

  • Latency between request and response, for AS-IS migrations:

For systems which only distribute information, a CDN configuration may be the complete solution. For systems who have transaction situations, maybe its needed to keep them at close datacenters or check architecture practices.

  • Latency within the application:

If there is a lot of available microservices, in many instances, different databases with decentralized information, and more than on cloud involved, beyond the client-server latency, there will be latency internally on the application. The latency has to be checked through architecture practices. Cache, queues, etc, can be used to prevent it from frustrating the end user.