Microservices Memoirs – Lachlan Evenson

Lachlan: "Everything was actually running on Air Metal on our own infrastructure internally. I think a critical component to this journey for us was we've never undertaken virtualization, so when you look at what I'm going to coin Cloud V1, it was high level objectives. We want top private/public layout. We wanted AWS and OpenStack. How that played out was we just forklift everything, all the tooling, all the applications and put them in advance. It was kind of job done. I think that was part and parcel for the day."

"Now another key concept to this Cloud v1 was with this waterfall, this kind of, "Hey, we're not of the Cloud. We want to get to a Cloud. Go and squirrel away for six to twelve months and come back with a cloud. Then the next six to twelve months we lift everything in. Now I'll go into what we did differently with MicroServices but just think about that kind of waterfall flow. We laid down Cloud V1. Basically, we had this reflection time. My boss and I were away at conference. We were reflecting over drinks about our journey, right? The cloud was internally successful. We'd been deploying applications on the cloud, and we'd met the criteria, but if we went and checked the polls of the development organization, we actually made their lives harder and that was counter to our mission."

"I'll have Mike actually talk about that a little bit. Mike was one of our key litmus tests and at the end of Cloud v1, Mike was like, "I don't like this. I don't know how to deploy. I don't know how to roll things out. I don't have a monitor. I don't know how to get things." Really, that wasn't satisfying for us. We've actually failed our mission. Even though the company had seen laying there was success, I think we had actually failed there. I think, if you were to call out a few things in why we weren't satisfied is we didn't service our customers. We basically took an operational point of view and reduced and mirrored all our operations in the Cloud. We solved the problem operationally, and we spend all our time in IS, not AppDev. We didn't go after actually solving people laying down apps, we said, "Oh. Hey, Mike, now you need to know where our load-balancer is, and back-end storage. You need fail over. You need to know horizontal scaling. You need to know logging and monitoring."

"Really what we should have done is given him all of those things without him necessarily knowing, and giving him the knobs that he could tune by creating a platform for that key experience for him. Fast forward to Microservices. Cloud v2, or going on to Microservices. I wanted to lay down some ground rules. Going in, what were we hoping to achieve with Microservices? The key to any problem is ... This isn't a technical problem we're trying to solve, we're trying to solve business problems. Now, I think you need to keep that at the forefront of your mind because when you hear the word on the street, you put your finger on the pulse, everybody's like, "Just go to MicroServices, it's all great." That might be the case but what are you trying to solve in your business?"

"I've been to numerous talks about containers, VMs, serverless. I advocate that if you have a process that is serving the needs of your business, why change it? That's a challenge I put to people. The other thing is put the customers first. Our customer was internal development org. Let's take them as an input and ask them first. Let's not ask them after the fact and say, "I don't care what you like. I'll tell you what you like." Let's have their needs be met because at the end of the day, they need to become our ambassadors. We can't tell them we have a good system that they don't want to use. It's not build it, and they will come. It's build it with them, and they will use it."

"Something else I wanted to call out was embracing the monoliths. There's all this talk of breaking down monoliths. If you're not [green field 00:04:42], you more than likely have some kind of monolith. Does it make sense to break up the monolith? If the monolith is paying the salary, control the tire fire. It's a known entity. The devil you know is better than the angel you don't. You can live and put bounds around monoliths. One of the keys to our success is that we didn't go after pulling apart the monolith. We actually said, "We have some new services. Let's make them successful. Let's be incremental here. Let's not try and rewrite the history books. Let's take something new, and start there and make that successful."

"You have to rethink everything. While you can go in and reuse some of your tools, you need to look at them all. You need to look at monitoring. You need to look at logging because the same paradigms might work, but it may not be ideal for how you want to manage at MicroServices. A few more little things I want to call out is that creating ambassadors. What you want to do is make your development all shine. You don't want a platform or the tooling around the platform be your star. I hear a lot of this but thank goodness, we have Docker and [inaudible 00:05:59]. They have vehicles but the business doesn't care if you run serverless Docker, proving that is. They care about the app how it's deployed and they care about metrics."

"About is the code good, our people worried about features, relevant infrastructure. Are we actually getting things to customs as faster? Are we actually delivering for people, features for AT&T and Skype and those customers? Staying in success zone. When I think of success zone, I think of, if you look at the market. It's moving very quickly. Stay within the bounds between what is successful in the market. Do not push yourself. We've been doing this for a year with containers. Containers weren't great for moving around, state for workloads on persistent disk a year ago. Let's not push the envelope and make that prerequisite, right? Stay where the market is working actively because it will mature and you can ingest it light. Slide that success zone with the market."

"Don't try to put yourself on the bleeding edge because you are setting yourself up for world of pain. Eat your own dog food. When you are doing, don't forget where the platform is aiming. Rewrite applications, let's deploy those applications as Microservices using the same tool of chain that we are making the application developers use. We should not think about things as operational tools. We should think about them as applications that have features and lifestyles. I really want to hop on Microservices are great and there has been great success for us but I just wanted to get some context around how we've been thinking than we're doing. The last piece ... I've heard there's different things on different schools authority, make the running environment a current authority. When you look at all these graphs of a million notes, it looks like a picture of the universe with dependencies and things everywhere. I've seen very mileage and containing that information in the document or putting it in a board. What I am trying to think of here, how about we just actually take what's running as the authority?"

"Let's just map what's running and use that as the documentation rather than putting strict bounds. Let's put tracing in, let's put the things that give us the current picture in all it's glory and make that a center of authority. Just somethings that challenge, maybe these aren't conventional but things that we've seen come to pass that are actually being fairly ground breaking for us and our journey. The catalyst, having drink with my boss. What we saw was an opportunity to actually strip this down. When we took a look at Cloud V1. What we'd ask developers to do, what they Opscode we've said, "Take Chef and this is one search example and now, please put your app in Chef. Okay, great. That's fine. What do I need to do, to do Chef? I need to write a thousand lines of metadata to run a young install." To a developer, that don't [fly 00:09:17], right?"

"We hadn't made it easy for them or create a [rapid 00:09:21] approach or made very easy for them to ingest but what we saw with containers is, just put how you run it locally. Put that command and create a nice little artifact and give us that artifact. I submitted that to the Dev Team. It was funny, I went away on paternity leave, I failed to say it to several developers. I came back, the repo was full and everybody was saying, "Where can we go to Prod", right? In two weeks, I got more traction with containers than I had with a year of asking people to do cookbooks and wrapping. Having that motivation being driven by developers was a really big catalyst. The other thing, the other side of that was if we didn't get ahead as a platform team of the developers. The developers were going to go and do it their own way."

"They were going to put Docker in Prod themselves. We knew it was coming so how about we get ahead of that and help them so that when they say we want to go to Prod tomorrow we say, "Data, please give us your artifact. We will actually submit that and show it, how they run." Mike can talk about that a little bit more but that's what we did. Now, one last Ethos thing for our team is incremental revolution, right? We basically just want to solve one problem quickly and then move to the next. We want to do it with a sprint agile model. You don't want to the say go into the whole and then six months later here is our WizBang [inaudible 00:10:57] infrastructure, too many of these infrastructure. Docker infrastructure, you want to say, I have one guy with one of these case. Can we do it for him?"

"Is he successful? What's missing? Increment. You want to sequentially solve problems because if you look at the world of making everything perfect. You'll never going to get there. You've got to just get to the next step. We made decisions, we made with containers, we got to buy in and developers were often rising. If I am to look at the counts that apply. I think the first one is off topic. Opinionated common pipeline and platform, right? What you end up wanting with Microservices, you want a consistent way to deploy Microservices."

"Bills from GitCommit, how does that get out? What is the workflow look like from the developers perspective? What you are trying to mirror is the Docker experience essentially. You want to be able to write your code, stuff it in the container, ship that container. The platform, make it redundant scale version and always find good things you come to expect. The opinion is what counts. If you are trying to have no opinion, you end up with a platform that's devoid of opinion. Devoid of personality I should say. There's no fingerprint."

"In early stages, about common pipeline you say, "Hey, man you want to write a job. You want to write a note. You want to write and go. You want to write and plenty. You want to write and bash." We will just pack it up and deploy it for you and each of these things have their idiosyncrasies. In light of being incremental, let's do one thing well. It turned out that 90% of that apps are written job. Let's go for the 99% or the 90%, not the one. Our opinion was to solve the largest user group problem not every user groups problem, first. That way, the pipeline in two months. I can get I ticket into the sprint. Please drop me, run NPM and I want to compile a note or whatever."

"That just becomes a feature at most of the pipeline which force into this modular elements. You don't want your developers walking around saying we use Kubernetes. We want your developers walking around and saying, we have a platform. They become meeting personal at Kubernetes but if it comes to pass that market staying in the success zone. WizBang, Dockers is actually doing a better job, we have the artifact which is a container. Let's just move the container orchestration to a different system and I don't need to tell Mike. Mike gets there. As long as the experience is the same. Mike can ask what's running it. I can tell him it's Dockers one, for example, what he says, but put yourself in a position to move because the market is not settled yet. There's no clear winner."

"Even if the next step is serverless you basically want a Microservice experience to mirror and to be consistent. The way we run apps is the same. They need to compute and they need some storage, they need some networking. As long as you can meet all those needs. It doesn't matter the vehicle that meets the needs. Just trying it out there. Our current state applies. Everything is in containers. All these Microservice are in containers. Whether people know that or not, not a big deal. New tooling, so our tool chain was only as strong as the weakest bits. We have to go back and rethink logging. We have to go back and rethink monitoring. We have to impart these things into a platform. Somebody Mike can expose some knobs hooks and get everything monitored and logged free."

"People want the easiest way to do something and they don't care how shiny it is. We wanted to just go out to making it easy for developers to get on board. You use our new tool. You will get logging and monitoring for free. You will get, we will have the pager. We will give you the graphs. We will monitor your JMX. We will do all these good things that gave them an incentive to let go of things that they would be doing Cloud V1 which was rolling their own set tools. Which really was our failing as a platform team. Before we go to some more questions. Is there actually any questions in the room? Not yet? Okay, I will answer a questions in a minute but I will just go over, dead bodies.'"

"I am not here to sell a prosperity message, this has been a great endeavor for us and it seemed a lot of value. We held a conference called Length, two or three weeks ago and there were two new product launches that were all container as Microservices. I don't know that they would have been launched had we have not have this pipeline to do it because the way that the developers could iterate within this platforms actually gave them the freedom to stand up fairly large on the takings."

"Trouble in paradise, these things aren't necessarily today's problems but just calling them out, I think that been said in one or two different ways in the last several hours I've been listening. RPC, HTTP over JSON you can Thrift, you could do Protobuf, you could do GRPC. Just stick with what is today's problem. If you saw the last Microservices Practitioner Summit, the guy from Uber got up and said, they move everything from JSON over HTTP to Thrift and somebody put a hand up at the end and said, "What was the difference?" He said, "Nothing."

"Right? Maybe it is not evident today, maybe scaling is better and I am not saying Thrift is bad, I am just saying go after the problems that are actually making sense. RPC maybe a problem for us now, it may not. Following these things out. Circular dependencies, they've been talked about today. Again, why not make the running environment the authority and have an in built way to detect these rather than ask people to look for circular dependencies because the environment has become end to power of end [fixity 00:17:43]. There's no way humanly possible. Why not bed functions metadata? Something that our system can pick up and actually follow your RPCs and say you have a circular dependency? I am an advocate of that."

"Consistent service contract points, this is really just saying, "Hey, make sure your API has an SLA and you move through features and deprecate APIs. Just keep extended [inaudible 00:18:13]. Each development team is now becoming a business that has customers. You need to bring customers along your journey. You need to be responsible for your API. I say, write the API first and adhere to the contract point that that sets now."

"I see various things here, people want to put clients. People want to write libraries to their service. You get crip in the library and you get crip in the client. Things aren't backwards compatible and your API is suddenly not. There's something in-compliant here. Make sure that you contract point even if you want to do libraries, clients or APIs. Make sure they are all moving into same pace. Otherwise, you are digging yourself a very shallow grave."

"Distributed tracing and circular dependencies. Great conversations even on the last summit about distributed tracing. I think that's neat to embed early and unknowingly put it in everybody's code. Don't ask people to do it. Give them a library or something that they import in and I will actually create what certain needs to figure out tracing. I think it would be silly for me to go to Mike and say, please input it. Please implement Zipkin. The next team, please implement. If I just say, "Guys, every time you start, you pull the same and all this is doing is stamping some metadata that each of your RPCs so that you tracing system." What you get for free like logging and monitoring is distributed tracing."

"The last thing I think that has already become an issue is latencies. You've move everything from memory to network. Some things have caches, some things don't. Some things take time and some things don't. It's about control. In a monolith, you are all in your own memory. Memory address basically all things in and out. RPCs don't work like that over networks. Networks can be problems. They can be latency issues. You need to think about that. What I've seen is people expecting memory based latency and replicating that as a baseline. These are all things that aren't necessarily today's problem for us but I see them coming down the pipeline. Let me take a quick look of where I am. Where my time."

"I will take a quick question here. "I like the idea of what is running the authority but I've heard quite a few problems getting signed off with [inaudible 00:21:01] is actually building the same as the original authority. I would keen here your thoughts and ensure imparity between systems during migration." Absolutely, this is the biggest problem and I don't know that I have a solution today. What I found is what's running is always is the truth. What you have documented, maybe the truth. Now, is there a way to reconcile what's running to what was meant to run? I don't know. I don't have a great answer for that but what I am proposing is implement, a big fan of your code is the documentation."

"Either build your code to be self documenting, putting documentation outside of code or that is a generated of the same thing as code. It's just room for scope or movement between one thing to another and you end up with just inconsistent stay. I don't know. I think it's something worth exploring and I think as we go down this path. Time will tell us too what is the better solution."

Brian: "Hey, Lachlan."

Lachlan: "Yes."

Brian: "This is Brian Kelly by the way. [inaudible 00:22:18] the last one anyway. We have a question in the room here. Have you found that there's a higher likelihood of better software construction in Microservices in Lithium because of the nature of the thing to smaller scope? Like we see a lot of people using actual models which is, it's more advance, it's more clean but it's very different than what you have done in modelling and then Pony running language, active base, et cetera. Do you find people, software engineers are looking at Microservices as this great opportunity to finally do it right this time and take a step up in their professional development?"

Lachlan: "Yeah, absolutely. We see a lot more experimentation because the cost to experiment is less. Because you could iterate faster, I saw somebody coming with a 3K statically linked go binary they other day and it became a race to the bottom. Their container was from scratch, statically linked binary. I am like, "This is really cool." I've opened up a lot of school I've thought. The only thing I say is, "Business problems first and we want to be opinionated." How about we move the things that are critical now and keep them in the scope of how we've been doing business until we've let the dust settle on to Microservice? If you want to experiment Pony or Go, that's fine. You are off the reservation."

"That's what we say now, right? You're off reservation. That's okay, right? Just don't ask for me to give you nice, smooth monitoring, logging with complete compilation of a nice ... You are not going to get the Maven or the Gradle experience like you get with Java or with Go right now. You want to build that? If you want to build that, that's fine too. I think another great asset for the platform is we build it in the open we've made it more modular so that somebody could submit their own PR to our system and we can build it in."

"Because we've actually taken and eaten now our dog food, our CI system is built on containers too. I've done several demos that we've moved, Jenkins was monolith and it host the [I-infrastructure 00:24:43] but we actually built everything in container so you can actually pull something in those that has Go 1.6 or 1.7 and I will compile against that and it's all down to a container in the Jenkins way of container and you could just go and pull those things in. It's nice and nested if you want to do NPM, if you want to do Maven, if you want to do Gradle, we'll support those things but at the moment we just have Java, DropWizard with Maven. That is the reservation now. Does that help?"

Brian: "Yeah, absolutely. One more quick question. That was great. As we were thinking about your last couple of slides there, there was the one that is common concern. That everybody interacting with the Microservice or consuming in API from another Microservice going to have to do. They are going to have to care about latency, timeouts, circuit breaking, whatever it is, security. We are wondering, is this going to cause the resurgence of aspect oriented programming? Or it seems like most people are gravitating more to a plain library as a way to solve that but it sound like cross cutting concerns that could be done with some of these aspect approach and these are some ..."

Lachlan: "Yeah, I mean, I think I could pass that to Mike as well and his set of questions. I appreciate that you bring that up because as I said, I am giving you the platform point of view but what we saw in the initial Microservices Practitioner Summit, was about three or four different skills of thought. We talked Google, the guys that wrote History X. The guys, Adrian [inaudible 00:26:24] was there. There wasn't one way to do it, but what I really took away was if you can do everything in your APIs, do you really need clients? If you've got Thrift or generic RPCs and you've documented everything, do you really need clients?"

"Also, library, should libraries become services? I think it was another question and a Google guy said, "No, Library should stay at Libraries." I think in the case of Google having one massive repo that everybody works on that, maybe that makes sense but in the case where you have despair of repos where people are working off of different sheet of music, that maybe standing things up the services, this is actually a better idea. Now, I think, I would say a person according, I think, we are going to have people go with client libraries, people go with library just import and off the operate object or answered approach or they are going to actually stand up new services with consistent APIs of that version. I don't know, the jury is out but as it is a journey for us, maybe in six months ..."

Brian: "Great. Anyway, we see behind your module ..."

Lachlan: "We would be like, "Hey, guys, this just didn't work for us."

Brian: "Great. Sorry, you're cut out there."

Lachlan: "No worries."

Brian: "I didn't mean jumping it in on your last comment. Speaking of Mike, is he behind you, right there?"

Lachlan: "Yeah, Mike is telling ... I am going to have him speak in a minute. I am just going to wrap up. I just wanted to put three or four key takeaways up on just to summarize. Microservices is not a technology problem, right? There is technology to drive it but your technology is a means to an end. It's not, your goal isn't technology, right? It's actually delivering product. The incremental, step, step, as you were saying right now asking questions. Does somebody run a right planning? I am fine with that. Is that solving a business problem? If the CEO said, "Thank goodness Lenny Penny, because it saved us." Maybe Penny is the answer. Do you get stuff down faster? Do you get that quality code? The more can you have this code over, can somebody else run it? I think should be a key criterias too when you are building services. Again, nothing against Penny, it maybe the right way to do things but I wanted to say stress, what I don't like is when people go out and blog."

"I failed at Microservices, a lot of I failed at Microservices is you didn't ... You went after technology not solving a problem, right? You went after Microservices to say that you were doing them. Not to solve a problem. As I said, if you've got it under control Tirefire that's earning money. Try more Tires on it and keep collecting your pay check. If the business is saying what you were delivering is absolutely acceptable to our customers. I think it's fair to say, then that's fine but technology for the sake of technology is never an answer. Keep it simple. Fail faster is always a good one, right? You go down, you acid test, it fails. Pull back, re-strategize, go again. If something wasn't meeting your requirement, don't just keep on hopping on it and keep pushing because what that just becomes is a negative experience for everyone. Microservice is non-intensive at experience than it is about technology. If Mike gets the ... I get my shit out faster, so sorry. If I get my stuff out faster and it's better quality and I am meeting my deadlines, then that's the measurable result, right?"

"Not making it harder for him. Keep it simple. We're engineers, we like to make things apparently complex. Keep it simple because it's three in the morning, you are not going to be creating yourself or making something that you can't debug. Never a cool thing to do. Opinion matters. In a world of every developer who gets hired, wants to write something in his own language or something that was conjured up last night before I went to bed. That's great. I love creativity but when you look at sustainability and pushing things to out consistently. You've got to draw a line. You've got to draw boundaries to be successful. You can't go in and say I have no opinion, do whatever you want and I will support it."

"I don't see that at Facebook. I don't see that Google. If you do things this way or you don't get stuff out. It's not. Bring your own language and we'll deal with it. That may sound harsh but I think you need to draw your line. Chase the MVP, not affection. Minimum viable product. What is the next goal to keep people engage in using the system and making them successful? That are my key takeaways."

Brian: "Those are very, very pragmatic points."

Lachlan: "Very pragmatic points. I wanted just to speak from a field of experience, not [inaudible 00:31:52] guy or [inaudible 00:31:53]. It's just, this is what we are seeing with our journey."