At Too Good To Go, Practical Engineering Keeps Food Out of the Bin

SE: Let’s get into the story here. Now, Werner, you had these two gentlemen on stage, but before they were too good to go, they were both at Endomondo, and so there’s a story here, isn’t there?

WV: Yeah, so can you just tell me what you actually did before Too Good To Go.

MK: Yeah, maybe I can start here, right? I can, I can say that back in 2009, I was still working at Nokia at the time, right? And GPS technology was an awesome new thing. We all knew these trackers from our cars, but then it became popular to track your running exercise, and I just have to say that was such an awesome idea, and I learned 3 Danish founders had started this company called Endomondo, which was all about kind of making people run. and making people become healthy and do sports and I felt that was just something that I wanted to be part of, right? It, it sounded so awesome because I like running, I like biking, I like staying healthy, and now this technology became kind of a lever for doing that, right? And Yeah, so I, in 2009, I got an opportunity to join this, and it was very early startup, right? There was one engineer and the three founders, none of them were engineers. So I got the chance to join almost without salary. And, great. What a great opportunity. Yeah. And, back in 2009, the iPhone was here, but it was not very widely used. So kind of my first task was to build a sports tracker for Nokia phones. So that was my way in and there was one other engineer at the time, and he was doing the back end, right? And so there was me and him and we have a few, we had a few students helping out as well. But basically, the two of us, we had to figure out everything, right? A mobile protocol, how, how does the mobile speak to the server. We had to design the protocol. We had to realize that in 2009, using HTTPS on mobile connections that went through all the mobile operators was actually not something that we could do, so we had to learn a lot of stuff. Yeah. And eventually we figured it out and the company started growing. Eventually, in 2012, we could hire a lot more people. And I think that’s when Robert came on board. yeah.

SE: That’s also when scaling starts to happen,

RC: hey? I came from a different background, so also engineer, but also coming from financial and insurance. It’s a very different beast than a startup, but I was like, hey, I want to try something new, you learn something, you want to go on something, and I actually started running a bit more, being more active after, let’s say, a slightly sedentary lifestyle for a couple of years. and I felt all this opportunity was kind of fit into it. And just joining a startup was, it was, it was fun. I mean, we were younger back then. a lot more time, more energy. Yeah. And I stumbled upon this job post by chance and I, they were running a I don’t know. I don’t think a lot of people run this production anymore, but Apache wicket system and that has a state, which will come maybe touch upon a bit later. scaling state on server side is, I can see that is not something that’s fun.

SE: It’s not something you do.

WV: I hope you did get paid.

RC: Yeah, I did, I did,

MK: we had to start to pay salary at that time.

SE: So both of you had some of those really interesting dive deep lessons early on, relatively early on, and then you’ve come together at Too Good To Go. Firstly, for all of us listening, what is too good to go and what’s the purpose of it?

MK: Too Good To Go is a company that was founded with the purpose of fighting food waste, right? It was originally founded back in 2016, maybe even early 2015. With the idea that kind of the original idea was a buffet restaurants, they were throwing out way too much food at the end of business. so the founders, they decided that, hey, some people would be willing to buy that food and for a reduced price and the restaurants would save money. They didn’t have to throw out their food. They could even turn their potential waste into a bit of cash and the, the planet would also benefit. So it was like a win, win, win, as we say in good to go, kind of the consumers get a bit cheap food, the stores get a bit of cash and the planet wins. and then they realized quickly that it was not only interesting for Uh, for buffet restaurants. A lot of food businesses have this problem that they like to have food available even at the end of business. So just before they close down, there’s still a lot of food, fresh food available, and instead of throwing it out, they could, we gave them an opportunity to sell it. So that’s what Too Good To Go is about,

WV: yeah. Where does the food normally go to?

MK: It normally goes to the bin, which is, it sounds crazy, right, when you landfill. Yeah, it goes to its landfill, right. And then, yeah, we have this way that is a win for the stores. They can They don’t have to throw it out and basically no one loses, right? This is one of the It kind of, this is one of the few things that we can actually do without any kind of, there’s no loss in comfort or anything, right? Everybody wins.

WV: I always thought to do good, you need to be an NGO, a nonprofit. But it turns out that there are many cases where he can do good. Have a good business and your customers are severely helped by it as well, so that you do good, but you still make a good buck yourself.

RC: Yeah, yeah, and I think, I mean, this is one of the approaches we, we also have is, our CEO Mindy Luge talks about is like if you are an NGO you are dependent on someone else, making sure you can operate. You need other funding from government or from, from the nations. And um, she says, and I think she’s right, and a strong believer in that, hey, you can make a win, win, win, you can make a business out of it. To ensure that you can continue to operate. Because if you, let’s say, let’s say we do this, we scale out, we can do great, but if we’re not here tomorrow because we can’t operate, we have no money, then what’s the point?

WV: Yeah, especially with uncertainties like USA and others where companies or organizations suddenly can no longer operate. Yeah, it’s a different, different story. But congratulations for your model.

RC: Thanks

SE: and I love that we’re talking about, waste not, want not when it comes to food at the same time as we’re talking about the frugal architect. So let’s start, let’s dive in to the, you know when you’re watching a movie and you know something blows up within the 1st 5 minutes and that sort of sets the stage for the story. We’re gonna start with that moment because you had the classic IT challenge of a business or an organization doing really, really well, and it’s actually a massive headache cause you’re doing so well, so it’s a good thing, but a bad thing but a good thing. What happened during the 2018 World Cup besides your team not winning? Tell us what took place and what that did in terms of your architecture and what it was and what it went to.

MK: So basically, soccer, like good old soccer is a big thing in Denmark, right? And there was a world championships in 2018 and at the same time, uh, I mean, our technology team in Too Good To Go wasn’t very big at the time, right? So I think we can safely assume that everyone was watching the soccer game, Denmark versus Belgium. Unfortunately, I think we lost, but that’s not the point of this story, right? But the point is that without us knowing, at the same time, our French country manager, kind of our leader in France, she went on national TV in France and France was not playing this evening, right? So they, there were actually plenty of people who were watching about too good to go and We crashed and burned, right? We, we got so much attention in France that our servers, they broke down, right? And Yeah, it was, I think I had been with the company for less than 2 months when it happened. So it was kind of like an eye-opener to me how, how unready, how not ready we were to handle such a situation. we crashed and burned, and I, if I remember correctly, it actually, it took us, some hours to get back up. We actually, we basically had to wait until the traffic softened until we could uh, get back up,

SE: thunderings.

MK: Yeah. And, yeah, I mean, it was a moment of pride, right? Because, hey, we are so popular, right, that our servers can be taken down. That’s, that’s awesome, but it was also kind of a big headache to figure out how we would avoid the same situation again.

WV: Yeah. So do we know why it happened?

RC: Oh yes, I think that’s, that’s on me. So I did join the company a bit before Morten. And so the initial server architecture was basically a PHP app running against a SQL database. there are some limitations that comes with that kind of architecture. you rely heavily on the database. There’s no state or caching naturally built into, to the language. It can be done, but it’s not naturally flowing as architectures and also the single database. So the moment you just, even with auto scaling on when you can scale compute, you can’t really scale the database and then suddenly everything just hammers down and at that point, you kind of lose it, even if you, if you manage to to scale compute, it will be stuck on database connections or services and they will be taken out and rotated and you can’t really do anything about it. So, yes, we kind of knew. So the people who built it, they had a great business idea, but

SE: Yeah. And you build what you can build at the time. This is, there’s this thing, premature optimization, premature scaling. This is the, this is why we need smart people, because it’s like, well, at what point do I make that decision? And unfortunately, the business doesn’t always let us know what they’re doing and where they’re doing it. I guess, Werner, you’ve seen that many, many times in many, many organizations that this is the unpredictability that the cloud is kind of there to help with.

WV: How did you get it back online?

RC: Yeah, well, actually, as Morton says, we did wait and then we came on. I think at that point we were actually slowly going over to AWS but not really using the infrastructure. So like a lift and shift, and if you do a lift and shift for, I think it was a C panel server somewhere, if it rings a bell with some people, yeah, which is like a single instance, everything running in once, which is great unless you have scale. So doing a lift and shift is not enough. We, we learned that during at Endomondo, like you can say similar problems, but also at scale. You need to think, you have a lot of opportunities even in 2018. Now we have even more opportunities. There’s like tons of things you can do. But, to get it up, well, we have to wait for traffic to go down and then, you know, what is, what took us down, right? That’s the next question. Is there any monitoring, not super a lot, but it’s very obvious we have when if you’ve ever seen the app, we like a lot of apps, we have a listing the moment you open. And that was a very, very, very huge SQL query. So, it will take you down. It’s only a matter of when so, the first thing we did was, OK, we need to do something about this. What can we do effectively, like fairly fast. And the first was like, hey, let’s just take the services that are the most heavy, that we can start moving over to another architecture. Possibly some caching, that would be nice. And then just move endpoint by endpoint to that new architecture. So, maybe for listings, it’s like you don’t need to think about what your business transaction, you think, you need to show your product and you start thinking about eventually consistency and and things that we may assume that everyone does, but if you just do a school project architecture side, it doesn’t really fit into that.

MK: I mean, maybe to take it a step back, we, when this happened, we already had enough signal from the business side that, OK, we are actually working on something that is successful here. So we knew that we had to be able to scale it and we had already started talking about what to do. And the team size at the time, kind of the total tech and product team in the company at this time in 2018 was, I mean, it was in the neighborhood of 10 people. so you can do a lot with 10 people, but you can’t do that much, right? so we, we had a lot of decisions to make, right? And one of the decisions we made was to, we’re going to build something. In Java, right? Kind of that was our first architecture decision and why it wasn’t the latest and greatest, but we knew it was reliable. We had people on the team who understood and we knew that we could recruit people in the, in the area of Copenhagen. A lot of people have that skill, so we kind of, that was our first decision, right? Our, our second decision, I think at the time was to Use as many services as possible from Amazon, right? We were already locked in on Amazon, so that was also an early, early decision. And we use as many services as we can, right? And build only what we have to build. And then kind of the third and very important decision that we made because probably as many other companies in this situation, we had this discussion, should we, should we pause feature development and rebuild everything from scratch and then kind of keep the business on hold for like 1 year or half a year or something like that while waiting. And we made the decision not to do that we’re going to do this gradually. So we kind of slowly started, kind of, I think first we did an experimental endpoint. Which was not very heavy on traffic just to make, just to prove the concept. But after that, we took kind of the most critical parts of the system. And rebuild them one at a time, right? So, basically, and I think we were super, super optimistic at the early days, thinking that we would be done in like half a year, of course, end of 18, right? And in reality, we retired the last line of PHP code. Two years later, in mid 2020, but it was OK, right, we, you got, you got the right, because we, yeah,

SE: and it’s hard. I mean, you’re, you’re sort of rebuilding the airplane while flying the airplane, and in my experience when people are sort of changing languages, the, the phrase how hard could it be always comes up, and then two years later you find out.

WV: Plus, plus the people who built the airplane aren’t there anymore. Hmm,

RC: yeah, exactly,

WV: yeah. So we need removing a single line of PHP and nobody complains, nothing breaks. It’s one of those cases,

SE: yeah. And this is, this is tens of thousands of processes running and things going on this is a big, this is not a small system that was running, it was, it was doing a lot of stuff.

WV: Well, you have to remember that at Amazon retail, we used to use Perl as our language for page description. And you know, at some moment you start to realize that there are no good payroll programmers. So you need to move away from it.

RC: At least, yeah, a write, write only language, don’t read other people’s spell code.

SE: So then let’s, let’s take the next step, and it’s interesting cause as you’re telling this story, it you’re reminding me there’s a, there’s a presentation we’ve had at Reinvent from the solution architects, almost from the start I think it’s called scaling to your 1st 10,000 users, or 10 million users I should say. And it tells exactly the story that you’re going through and then the steps you take, and it’s interesting how folks have to, you have to go through those phases, but you had, you had one that I think is a, a lot of, companies and software providers want to get to, but not all get to, which is suddenly you’ve got to go international, and again, that’s one of those amazing gifts of wow, we’re going global, but it’s also, oh my goodness, we’re going global, so give us some context of what was going on and how you tackled it.

RC: Yeah, I think so, as, as Morten mentioned, in mid 2020 we retired the last piece of PHP code. Great effort, guys. we’re also launching features. So it went well. I will call it a success. We were just optimistic about the timeline. But then it’s like NSA, hey, we’re going to go global, we’re going to go to the US and they’re like, we were like, OK, US was, what does that mean for us? Well, that means Latency. We were in the European Data Center in Ireland, which I guess a lot of European customers start off in and then The latency tests were a bit poor from our mobile experience. So what do you do? You have a You have an architecture, you have servers, they’re not built for multi-region. So, how do you scale that? That was kind of like, what is the experience we want to give to our, we want to, the users, both our consumers and partners to have a first-rate experience in the US. And how do we take that challenge, so we, we ended up with a solution where the application itself didn’t really need to know a whole lot about being in multiple regions, but we were simply route so a you have a home region and then there was a single component that knows how to route or fetch data, no matter where it is.

WV: So you do want customers to migrate between let’s say you’re you’re European good to go. And the US one, it’s not that you have a clear separation between them.

RC: No, so it is, and then of course, when you build for 2, we also build for multiple things and also expanded to in Australia later on so, it’s basically that you can go as a customer in the US. If you just fetch the the feed where the data is in the US it will just stay there. But if you have to interact with some data that’s yours, we’ll either have to fix that relevant data or route the route the entire call to your home region. And that is built into it and something that we have standardized. So, each developer just needs to remember they need to root or, but otherwise don’t really think about it.

WV: So the customers are worldwide, but I assume your offerings are extremely local.

MK: So basically, uh, we don’t, now we do a little bit, but back then we did not offer delivery, right? So the food offerings are super local, right? Because the consumers, they actually have to pick them up at their local store. so they have to walk to the store. So typically you will buy something that is within a few kilometers from where you live or where you work or something like that but I, some of the thinking behind not running completely separated architectures was rooted in business, right? During our expansion in Europe. every time we open a new market, a new country, we have a kind of a cold start problem, right? We, we need to get some stores onto the platform and we need to get consumers. And we learned that the tourist use case is actually not nothing, right? We got a lot of, in the early days of every new market, we get a lot of help from people traveling, right? So, we did not want to miss out on that one, right? We wanted to, we wanted tourists visiting the US from Europe to help us drive the initial growth, and so on, right? So there was a business reason behind it. And we have not regretted that one.

WV: To, from the vendors or the restaurant’s perspective. do you have to go out and find them, or is there something where they come to you?

MK: Yeah, we have the, we have sales teams in the, in every market where we operate. and typically, we start from no awareness at all. that is why, and then kind of we have the outgoing sales teams and they reach out and then eventually, when Too Good To Go becomes more famous in the market, then we start having kind of the inbound. requests and then kind of that also kind of helps, helps the growth machine. So

WV: very cool. Excellent.

SE: And so one of the things you’ve spoken about in terms of this architecture and some of the choices you made was boring, quote unquote, technology. and not in the pejorative sense, but to help us define boring for us and why you like boring.

RC: Yeah, so, so actually we like to be boring but not stale, a bit more. So, and let’s say in the case of Too Good to go, right, you have some people, there was a young entrepreneurs, they’re not technologists, but they got a great idea and got a really good start. And like if you then come in and say, hey, we’re going to change it out, I’m going to use the latest and greatest. Then I’m actually betting that business. I’m taking that bet and I don’t know how that’s gonna go. They don’t really care about technology in that sense. Like we as technologists, we care about technologies, but I also like something true and tried in that sense that It’s a safe, right? You need to know how to use it. It’s not enough that, hey, now we have the newest language or if it’s a JavaScript, you have a newest library number 1000 this month and you’re going to change everything and someone deprecates it. You have to be a bit boring. You have to, you’re going to live with these choices. Software, like I was in a in an insurance company, there was mainframes around. There was PL one, and for those who doesn’t know what PL1 stands for, it’s programming language one. And this was in the, in the 2000s and so it’s like your choices will You, you have to plan for long term. So, don’t bet the business just because you want to try something. Be a bit safe, but also make sure that you continue on because technology to progress and as there’s there’s, I forgot who the famous quote is, like, technology is progressing at a faster pace than you like never had. It’s always going faster and faster. So make, make the smart choices and then maybe if you can avoid really over committing to something. Then you have a choice forward. So, that’s what I mean with we mean was boring, you know. Choose something that will

WV: work. Are most of your developers still in Denmark? Yes,

MK: we have the bulk part of our development team in Copenhagen on site, and then we have a reasonable sized team in Paris, and then a little bit in Madrid, right? But try, we’re still trying to grow out of Copenhagen. I think we are, we are around just, just below 200 people now. so, and, and as, as Too Good To Go has become more and more famous, then we are still able to attract good people. in the company, but I mean, eventually we will run out of, talent in Copenhagen and we will have to expand further.

SE: And it’s interesting too, I think, I think as you, as you’re growing, you’re thinking about doing this again frugally, and just to give a sense of scale, now this is a platform that has over 81 million registered users worldwide, 145,000 active business partners. So, there’s a lot going on the system. And one of the philosophies you have is, always ask why. Help us unpack that, cos it’s interesting as you’re growing and scaling and clearly moving quickly, different countries, it’s kind of like, well, we’re doing stuff, always ask why, sounds like an interesting sort of, pause for a moment.

RC: So what we want to do is like when you, when you do something, it’s like, don’t do it just for the sake of doing it. It’s like get down to the core. Why are you’re doing this? Why are you implementing this feature? Why are you taking this technology choice? like, are you doing it just because you think it’s fun or because it’s gonna help the business or it’s gonna help us scale. Just like there needs to be a reason and it needs to be a good one because every time you take a choice, you also decide not to do something. and we do have a tendency, we are as humans, biased by biased for action, but sometimes you do like, maybe you shouldn’t or maybe you should just cut down. It’s very easy to get excited by, I can also get excited by technology, but I tend to like, OK, then I’m going to play around with with a laptop rather than saying, let’s change the entire back end infrastructure. So it’s about you know, why are you doing something? hm

SE: it’s about there’s that that purposefulness I think comes into it and that’s, that’s a challenge. I know a lot of folks really struggle with it because like, like you say, things are moving so quickly, like, everyone’s talking about LLMs and Gen AI and there’s this new language and that new language and this framework and this like and what you’re suggesting here is it’s important to just sort of like take a beat

RC: so

SE: so if you be on the on the ball for the right thing.

RC: Yeah, so if you take LMMs and AI, it’s like you’re also like, what problem are you trying to solve, right? If someone goes, goes to tell you, I’m not saying that’s definitely problems you can solve with AI LM. It’s like, hey, let’s use AI and like, OK, what problem do you want, what do you want to solve? Because there’s also 1000 choices just LLMM just like, like exponential choices. So, what is the problem you want and then we can find out, like, we don’t need to maybe try every single limb to try something, but like what’s the cause of the problem? Where, always when I have a problem, like, how do you see, hey, you come to me with a problem. And my first question would probably be like, so once we have solved this problem, how do you, how does your world look? How, what’s going to be different? What does your workflow do or what are we going to do differently? Before we start even thinking about building anything like, so, and also a lot of the times, you never get to the full le, you get Well, enough that you get the value and then you continue.

WV: I do notion that my, most of my customers are overwhelmed. I mean, there’s 10 new models every week and so there are so many stories continuously that it’s a hard time for customers to make a choice. Do what we need to do. Do we really need to? Then, I really like your approach. What’s the problem that you try to solve with this? Yeah, so I think you guys are on a very, very well path. Many of my customers would like to push a pause button. Yeah, we’re living in exponential times, and we have a hard time tracking it.

MK: Yeah But I think Robert and I, we, we both also, we feel a lot of pressure to Kind of just as anyone else, I assume that right in these AI times we are also scared of like falling behind, right? So it is like, I totally get that fear that, that is all over the place. But I think we are, we are taking an approach where we experiment also in areas where we, where we see, OK, this has something to it, right? This is So we are experimenting with, kind of all our developers have access to AI tools that they can use for programming. We haven’t decided on a, on a specific one. We, we let every kind of, we think this is the early days, right? So everyone get allowed to experiment with what they do. And then we have kind of learning Slack groups and learning meetings where we try to tell each other what works and what doesn’t work and kind of learn along the way. and that works fine.

WV: That’s our deal, I think, because many companies really suffer from FOMO, the fear of missing out. 3 years ago, customers would ask me, what should we do with blockchain. Nobody asks that anymore.

SE: Yeah, I heard a short answer on that one too.

RC: Yeah, I think we sat that one out, fortunately, so.

SE: it’s interesting too how the fundamentals are the things that, that don’t change, and as you’ve been speaking, I’ve been thinking about some of the aspects of the cloud that we’ve been speaking about for many, many years now that, that you applied in your business, elasticity, the ability to grow and shrink based upon demand, and go global in minutes. Now, it didn’t take minutes, but you went global very quickly. I’m guessing you didn’t fly to the US to tour data centers. You just got on the console and clicked, US West 2 and away, and then you did your AP Southeast 2 and you were in Sydney, you know.

MK: Simon, just to add to that, kind of if you look at the, if you look at the timeline, right, even if we had wanted to, we actually expanded to the US in the middle of COVID, so we couldn’t even get into the country, so the people who went there to kind of start the commercial side, they had to fly private planes and spend two weeks in kind of isolation in a week in the Caribbean before they were allowed to get into the US to start up the business. So it was for sure interesting times to open up the US so.

WV: Is there a difference between your customers in Europe and the US? Do you see, do you see different patterns?

MK: Our business is super local, right? It’s like because you have to pick up the food. so, one of the things that we had to learn is that in the US people travel longer distances, right? In, in most of our European markets, our customers, they walk or bike to pick up the food, right? In the US they drive cars, right? so, there are some differences, but in general, even in the US people eat food, they like a good deal, and the stores hate throwing out food, right? So there are more similarities than there are differences. OK, cool, yeah. But the thing about the transportation to pick up the food that was an interesting one that we hadn’t really considered before, before going to the US. So.

SE: Ah, it’s super interesting and it just such a great story, I think of as, as, Werner was touching on at the start, the ability to do good, have a thriving business that, that sustains itself and nourishes and sustains others is fantastic.

WV: And thank you for doing something extremely special that helps a lot of people. And eliminate waste. I remember you guys had a CO2 kind of thing in your presentation as well, sort of there. So you’re not only actually doing good with food, there’s a carbon impact as well.

MK: Of course, yeah.

SE: Less is less, isn’t it? I think food waste, I think it was the stat was food waste is 10% of all greenhouse gas emissions.

RC: Yeah, that’s the, that’s the one. I’m desperately trying to find the presentation to get the numbers right here.

MK: But it is insane to think about, right? Because it’s not food, it’s like, it’s just the food that we waste that that accounts for up against around 10% of the total carbon emissions. So yeah

WV: Yeah, so we recently had 3 days of strikes of people picking up garbage, and the city of Amsterdam decided to put all that stuff in one place. If you look at what the garbage is of a city of 800,000 people in 3 days, it is shocking.

RC: That is, yeah, it is.

WV: So wait, enough problems to solve.

SE: Yeah. Well, that’s the nice thing, we get to keep building and solve problems. Werner, thanks always for, connecting such amazing, organizations and bringing the stories and the context as well.

WV: My pleasure, and thank you Morten and Robert.

RC: Yeah, thank you,

SE: pleasure to everyone out there, until next time, keep on building.