Discussion: Mike Ottinger and Lachlan Evenson on Adopting Microservices at Lithium

Mike Ottinger (Lithium)

Description

Lachlan Evenson interviews developer Mike Ottinger about what it was like to adopt microservices at Lithium, both the good and the bad.

Transcript

Lachie: "This is Mike Ottinger. He is my esteemed partner in crime. He's actually been not only a guinea pig - hopefully we've humanely tested things on him - Mike had hair when we started with microservices. Mike has been really accepting of the change and I really wanted to get Mike in to give you guys more context about how he's seeing this transformation and give you his take on things. With that, I'll open it up to Mike."

Mike: "Great, thanks for having me Lachie. Very excited to talk about the experience that we've had on our team with microservices and just what you've been talking about earlier. We've got a history with microservices and it has followed a narrative of this is our platform, we've adhered to it, now it's definitely become- you've taken care of a lot of the utilities and the plumbing that's necessary for us, but it's also given us the ability to work within that sandbox and be able to innovate and iterate quickly through containers."

Lachie: "Right. I have a couple questions here. Let's step through them. What does microservices mean to you?"

Mike: "Sure. Everybody has to realize that we think of microservices in contrast to the monolith. We've seen microservices as a way to keep the monolith where it is and be able to break out of that a little bit and work faster. Our deployment times within the monolith, it's called LEA, are monthly. We've oftentimes found that we want to iterate much quicker with our customers on features more so than just on monthly releases. One of the first things that's been about microservices for us is that we can have an idea, we can implement it, and in half an hour we can have it out to production if we pass through our pipeline of tests."

"As Lachie had alluded to earlier, we are by and large a Java shop, but we've found that with microservices and with taking on the infrastructure pieces on our own, we can play around with different technologies. In fact, we've got services now that are about 50% in node and 50% in Java. Some of the node stuff we definitely have our own ways of monitoring these things and we fill in the gaps in terms of what's not there from an operational standpoint. Microservices are great for us just to give us that playground to work quicker and to be able to play around with tools that fit the problems easier."

Lachie: "I think that plays into the next question, which is how has microservices changed the way you work? You came in, and when you started you were working on the monolith, right, and you've gone into the microservices. You have a good baseline context of both sides. How have they changed the way you work, both for better and for worse?"

Mike: "On balance, it's been an improvement. The things that have changed us from really day-to-day is just ensuring that it's fun to work on a brand new service that nobody's using yet because you can re-factor ideas and just on a whim go in different directions technically. Where things start to get interesting is when the monolith has a dependency on it or other services have a dependency on it, and now you start to get yourself locked into a contract with client so how are we able to change things in such a way that they are backwards compatible? For me as a tech lead it's become part of another PR, a poll request, that we do. Is this fix going to break things? Thankfully we've learned that without having any catastrophic issues. We've had a couple of times where we've deployed things and it's like that didn't work. Revert. Let's take a closer look. Let's not remove an API route without putting some sort of shim in place to ease it out. There's definitely those types of things."

"What it's done also is it's instilled in us a little bit more of a holistic model in terms of how we own our future. With the monolith, there's already a group of people that manage it and take care of it from an operational standpoint. With us, we have the tools in place as Lachie mentioned for instrumenting and monitoring but it's up to us to put in what makes sense because we're the ones that are going to have the dashboards and we want to be able to see how our services are being used and be able to look for the smoke before it turns into a tire fire."

Lachie: "Absolutely. Have they actually made your life easier? Is this something that has been a worthwhile endeavor?"

Mike: "Yeah, I would definitely say it's a worthwhile endeavor. I want to be pragmatic here. We have to look closely at every feature that we do and see does it make sense as a service on its own? Does it make sense to put it into an existing service? The micro in microservices for us is sometimes in quotes. We strive to not only just make new services for the sake of making a new services. Maybe we can touch on that in terms of what are the challenges. You can pull in four different engineers into a room and you'll get four different answers on what they consider to be the ideal state for a service. We really try to be pragmatic. To be perfectly honest, we're still in our infancy with some of this and we don't know that some of the services that we've written we may find that we're always making certain types of changes together and having to deploy these and affect the larger services. Maybe it does make sense that we start to move things out into their own service. It's like the macro level of object oriented programming. You keep the things that change together together so you don't affect the larger system."

"Those are the types of things that we're encountering. All in all, the services has been a huge improvement. It's allowed us to work faster, to add better features, much more so than the Herculean effort to get our monolith into shape where we could work this fast."

Lachie: "I think you brought up something that was really interesting. I think its important on your journey too, it's an organic, there's no hard and fast, it's a very organic lifecycle with what services are. There's no right or wrong way. Libraries, services, client libraries, how big it has to be, multiple functions. I think the key is that we provide flexibility for you to okay let's put that out. Let's just whack an API on that piece of function already that is in another service right now because 10 other services need that."

"What are some challenges associated with microservices? Hidden things, technical, from your engineering perspective."

Mike: "What I had alluded to earlier is striking that right balance between too big or too small of a service. That's definitely been one. Also just ensuring that we do have backwards compatibility in terms of the operations of a service. Those are really the two challenges for now that are- and they're not even really that big of challenges, they're just things that we need to consider. It's more criteria for what goes into a new service or what stays in an existing service. Off the top of my head I think that those are the two big challenges."

Lachie: "Okay. Pie in the sky, things you see on the horizon. What are you hopeful for? What do you see coming down the line?"

Mike: "We're definitely just beginning to embrace containers. I see a lot of promise in that for the work that we're doing. Some of the more pie in the sky stuff for us is a more mature diagnostic process so that when we have a request that goes through 5 or 6 services, how can we accurately trace the lifecycle of a request through all the services and look for particular issues?"

"Some of the other things would be the server-less technology that you hear of. Lambda and stuff like that. That feels to me like we're just getting into containers and these are very exciting things. Lambdas and that type of stuff is alien technology at this point. Those are exciting things. It's exciting to know that the landscape is evolving even further but to what you had alluded to, I feel like we have to find the technology that works best for us."

"In some degree, we've put blinders on a little bit. There might be something really cool that comes out in six months but if we're just constantly chasing that carrot, we'll never really get to be able to mature on our tooling here because we're always looking at the next best thing."

Lachie: "Yeah. That finishes up my set of questions. Is there anything else you felt worth mentioning?"

Mike: "No. One of the things I wanted to bring up too. This is maybe stepping a little bit outside of microservices. I also have quite a fancy and more of on the front end technologies. I'm very excited about emerging JavaScript technologies like Angular 2, and React. It dawned on me a while ago that we've also been able to make strides and efficiencies with our monolith due to the use of Angular. We've been able to move a lot of things out and into a UI. What that's done is it's allowed us to surface up APIs so we really think about functionality from an API perspective. What that's done in turn is it's informed us in our decisions on how we would make microservices that need to serve those API. We've achieved a couple of Angular straight to microservices via cores and [inaudible 00:11:15]. It's very cool stuff. It's kind of connecting those two worlds together in a way that's really exciting. It's a victory in my book when we can write a non-trivial piece of functionality and actually not touch the monolith at all. Just skip it."

Lachie: "Yeah. Cool. I hadn't actually thought of that. I think that's neat that you can tie those two things together in your journey."

"Thank you very much, Mike. I'm glad to have you here. I think it adds a lot of context around the stuff we're saying. We'll open it up. Is there any questions?"

Mike: "We awed them into silence."

Lachie: "We awed you into silence."

"Yes, one second."

"Worst war stories?"

Mike: "Worst war stories.
"
Lachie: "Everybody likes war stories."

Mike: "Nothing real cataclysmic. Maybe bad configurations where we have- trying to think. Deployments that just don't go well at all. Something dies in a very critical phase of the deployment and it leaves things in a funky state. Actually that's a great thing that I haven't really talked about is how important it is for us to have a rock solid deployment pipeline. That's something that we've got and I'm very thankful of. Some of our war stories have been when that wasn't as sure. For us, having the ability to be able to make an artifact of a docker image and after having run through tests and essentially being able to walk away is important to us. And also being able to roll that back. I think that most of our war stories have been really on the deployment front, which emphasizes how important that is."

Brian: "Can you guys hear me okay?"

Mike: "Yes."

Brian: "Great. Sorry for the delay in getting back on. Keying off the worst war stories, what are your- microservices is talked about with such benefits, but what about your pet peeves? Do you ever yearn for the monolith and say, "You know what, I wish I could have a 7 gigabyte jar file I could deploy right now?""

Mike: "I'll say not really, but we don't work with 30 services. I'm always mindful that I feel like we have a fairly good process for how we work with our services. Right now we've got about 4 or 5 in our feature. That's just within my group. I think that across the company we probably have about 30 services, but a lot of our services really only have to interact with our own stuff. There's other services out there that we have less to deal with. When we get to a point where we have a real intricate network of inter dependencies, that's where things will get interesting. That's stuff also that Lachie had alluded to. We really have to be good about understanding who is dependent on whom because it will get tricky. Having a very sophisticated way to diagnose problems within services will be very instrumental. Right now, thankfully, we're not that far along enough where we can't figure these things out for ourselves."

"Right now, it's all relatively rainbows and unicorns."

Brian: "Got it. I have an interesting divergent question. How has it affected your recruiting? I'm sure both of you are involved in interviewing people. You're going to come across engineering candidates that have worked at microservices-based companies but probably a lot of them have not. Are they excited by it or are they perhaps scared that they might be the sole contributor to a microservices and the spotlight will really be on them to not screw it up. Is it a net positive?"

Mike: "I'll be perfectly honest. I have not really interviewed people through the lens of microservices, per se. I talk about the tooling that we use, obviously Java and JavaScript. Then, when people come with Chef and Docker and maybe a couple more devops style tools, those are definitely pluses, but I feel like for us we can instill the best practices and the processes that we need. We can just instill those as on-the-job training. To Lachie's point, a lot of this is not a magical set of technologies. It's just a methodology for how we get features out to our customers. A lot of that is, "Hey everybody, when you're doing PRs, look for the obvious stuff but also just think about how this thing is going to run when we deploy it out there."

"It's all relatively new for a lot of us. We're all learning together."

Lachie: "I have one thing to add to that. As I said in the earlier slides, don't think of microservices in a vacuum. There are a lot of things that are changing at once. I think being a cloud-native app developer is probably something that's more valuable than understanding microservices, per se. Cloud native is understanding the infrastructure that your app is born on. Understanding the idiosyncrasies of what to expect from a cloud is interesting. But on the flip side if we build a platform that you don't have to worry about that, you just have to write code that is horizontally scalable and we'll make sure it's run on three different machines, it gets attached, you get the storage you need, you don't even really need to know about the cloud at that point."

Brian: "Kids these days."

Mike: "I think that actually, the more I think about it, maybe this has been more of a subconscious criteria that I've had. I'm looking for people that are curious. It's not just magic that got the bits in the box. They actually want to log in and see how we deployed it and how it sits on the servers. Getting people that are interested in AWS and Open Stack is definitely a plus."

Brian: "I think mechanical sympathy extending as far as you can go in a virtualized cloud native environment."

"Speaking of cloud native stuff, everyone is talking about uni-kernels. Have you guys looked at that at all or is it one of those things that's going to be in [inaudible 00:18:31] for another year before anyone does anything serious with it?"

Lachie: "I've looked at unikernels quite a bit. I don't have too much to add to them at the moment, although I don't feel like as I said staying in that sweet spot. I think it will be interesting to see how Docker folds them into their story because I don't think we've seen that come to publicly pass at the moment. Obviously they have acquired a company that is working with unikernels. It will be interesting to see if that actually comes to pass in the whole dock cloud movement."

"With any new technology, you've got to be able to tell a story and show a tool chain. Unikernels freestanding, does it add much value? No, but as part of a tool chain that delivers some intrinsic value, sure. As long as they're delivered with some really great story or use case, I think that will be maybe compelling. For some people, it's still early days."

"One of the biggest things as a platform team is changing, being incremental, changing operating systems is a hard increment. That's a big incremental step. It's packaging, it's security, it's so many things. Again, what's the business outcome? It's like people saying, "don't run containers on VMs because you lose the value of containers," and I say, "well, if you're app's horizontally scalable and each container equals some piece of performance, then surely you can horizontally scale and get the same performance without being on a kernel that's not between a hypervisor and hardware."

Brian: "Right."

Lachie: "Is unikernels today's problem? I don't know. For some people, maybe. For embedded systems, potentially. That's all I have to say on that."

Brian: "Got it. Great."

"We had a question as well come up about security between services, inter-service security. Mutual certificates, passcodes. Anything like that or do you just let it rip?"

Mike: "You broke up a little bit."

Brian: "Okay, I'll repeat the question. How do you handle inter-service security for authentication or authorization? Do you implicitly trust any request that's behind the edge services or behind the front firewalls, or is there requests authenticated the whole way along?"

Lachie: "Who asked this question? It's almost like I paid them. Can I take this one?"

Mike: "Yeah."

Lachie: "One of the things we wanted to [inaudible 00:21:10] earlier was like the whole getting in front of the Docker movement, so having your production solutions in containers before the devs went and made one of their own. With security, what we did was we built a really sweet system around SSL that we're still working on. Everything's rest at the moment. [inaudible 00:21:32] HTTP. Adding TLS to that is actually not a hard feat. When you break that up, what we actually have is, maybe this is a longer discussion, but essentially every service that comes up talks to [inaudible 00:21:45], it asks for a certificate, it gets issued a certificate. We do client and service side TLS certificates. There's a TRL in vault that's looking for certificates. The certificate is valid for the lifetime of the service. If you've ever looked at the PKI backend in the [inaudible 00:22:05], via API you can say give me a cert. I am Lachie instance 1. I am Lachie instance 2. I am Lachie instance 3. You connect whether it be via thrift with TLS or HTTPS with JSON, whatever you want to do, you've got the API call a way to get the certificates."

"This is a lot better than hard baking wild cards in every container and just trusting them. It does mean that you have to think about failure scenarios. If that API goes away and you can't get certificates, does that mean that every service doesn't work or do you have some kind of dry foul back or do you have some kind of catastrophe, disaster recovery?"

"When I sat down and spoke with a gentlemen on Mike Ottinger's team actually hit me up about this question and I had a fairly valid answer for him which was is the interface, write to this, just put in a failback. He said great, you've already thought about this. I think we could talk about how we've implemented that in another session, but we do have TLS between services with client and server based certificates with CRLs."

Brian: "Wow. For the audience, that was not a prepared question."

"You said something which I didn't want to interrupt you. You said the certificate is issued for the lifetime of the service. Could you expand on that? Don't they have an expiration that's finite?"

Lachie: "They do. We just have refreshed tokens that can say give me a refresh on this thing. We're actually refreshing and saying kick the lifetime out on this or [inaudible 00:23:53] on the certificate while I'm alive. Re-key this thing. There are a number of different ways you could do it. Or you could give a TTL on the certificate that lasts a day and service. Whatever you want to do, whether it's expirations or key-baked. You can do some cool things with vault and actually make it reissue a certificate. It's not a science right now, but there are those possibilities."

Mike: "Just on the application front, for us, we use teraform for building our stacks. Teraform for us basically maps in the certs for us to use on AWS where we don't even have to think about it. Using HTTPS, we've really tried to strive for HTTPS everywhere. We still have some pockets of our infrastructure that aren't quite there, but even behind the firewall HTTPS everywhere. With teraform and the work that Lachie talked about, for us it's as easy as following a law. You just use it. There's really not much that needs to be done."

Lachie: "Looks like another question. Initial service delivery of the certificate to the service? We're actually using [inaudible 00:25:11], you basically an encrypted key valstore. We do a very basic two-legged [inaudible 00:25:18] where we have one system give an app ID and we have another system give a user ID. The next instatiation app, those two map to an [inaudible 00:25:33] that allows you a subset of back-ends and vaults. But, you're speaking over HTTPS to vaults, which will deliver the certificates. I hope that answers your question."

Mike: "In our CI, I can speak on our platform. We have an app ID and a user ID that's embedded as an environment variable, an encrypted environment variable in bamboo for us. That will actually start the handshake with vault. We'll pass those in and say this is the bamboo user, I want to get an AWS set of credentials using these. That secret is introduced there. I personally don't know what those secrets are. They were set up by these guys. They won't tell me."

Lachie: "We don't know. I think we've used a mixture of KMS and AWS and we also use [inaudible 00:26:35] secrets. To get that lock and the key from two different places with two different sets, tie them together to give us access to the system to pull certificates and ask for certificates, or encrypted key vals."

Brian: "That is a very elaborate and sophisticated answer to security. Most companies there's like, they've gone to microservices and think oh wow we should probably do security."

Lachie: "One of the things we laid down as a team is we wanted to have this framework straight up because we knew the initiative was as we fanned services out we always had to deliver HTTPS on the backend or some kind of TLS on our PCs. We thought about it. As we've moved a lot of our secrets infrastructure into vault, we've found out that there's a PKI back end that could actually service this type of request. We've been experimenting with that."

Brian: "Great. Alright guys, it's been wonderful. Thank you so much Lachie and Mike. We're going to have to wrap it up.
"
Lachie: "Thank you, guys. I'm an open book so either Tweet me or figure out how to get in contact with me. I'd love to help anybody on their journey or answer any questions. Thank you very much for having us and thanks for the great day."

Expand Transcript

Stay in the Loop

Keep up with the latest microservices news.

Simplify and streamline microservice deployment.

Try the open source Datawire Blackbird deployment project.