Hello fellow concerned developers, or, if you are not yet concerned, while it is not the crux of today’s entry, I hope\nthat by the end of it, you might get interested in becoming one and/or find out how you could do that on your own and\nwhat it actually means for us. Still, the focal point of this post is to fill you in on the architecture of Application\nCockpit (or “App Cockpit”, as we call it in private), the newest tool AUTO1 has brought in for its concerned developers.
\nThe “why”
\nHowever, before dwelling on the “how” too much, let us pay homage to propriety and start with the ‘why’ of the whole\nmatter. The main reason for the emergence of the Cockpit stems from the plethora of tools AUTO1 has adopted to help us\ntrace, measure and monitor our services, not to mention alerting us when things go south. This opulence often means that\nif you are looking for a specific bit of information, whatever the reason, chances are the data is there somewhere, but\nyou need to make a significant effort finding out where to look for it first.
\nTo achieve all that, the Cockpit needs to pull data from multiple sources, aggregate it and present it in a comfortable\nand meaningful manner to the users, not to mention being fast and efficient about it, which leads us to the heart of the\nmatter - the architecture of App Cockpit.
\nFinally, the “how”….
\nWhen designing App Cockpit, we were striving for what probably every software engineer worth his hide can recite even\nwhen woken up at 4 am in the morning - we wanted the architecture to be “open and easily extensible, yet simple and\nrobust at the same time\", preferably allowing polyglot extensions to let every team in the company contribute and share\nwith others if they came up with a cool feature. In order to achieve this, we have split the Cockpit into 5 logical\ncomponents distributed in 3 different layers, as depicted below, with the descriptions to follow.
\nData Collector - the component responsible for fetching data from multiple external sources, parsing, aggregating,\ntransforming and sending it on its way to the data sink with a tiny stretch it might be called the ETL of App Cockpit
\nData Sink - a simplistic component with a one-track mind, whose only job is to receive data in a well-defined\nformat from the Collector, occasionally perform some simple and generic operations on it (like basic validation and\nparent id resolution) and persisting it in the data storage
\nData Storage - a simple storage that holds the current snapshot of App Cockpit data
\nData Provider (aka Backend For Frontend) - exposes an interface to serve everything our UI needs, both serving the\ndata “as-is” from the storage as well as applying some additional logic to further aggregate and transform it for\nspecial purposes
\nUI - yer plain old Front-end - by design contains no logic, it only displays the data as provided
\nSweet, but how about a model?
\nThe specifics of the model run outside of our scope, however, getting it wrong could severely hobble the architecture,\nso it is a story in and of itself, which will have to wait its turn to be told in another blog entry. For our purposes,\nwe will only briefly touch upon it, mainly to establish the domain.
\nTruth be told, App Cockpit is pretty much a simpleton, since its world consists of only two kinds of things: Readings\nand Resources and everything it does revolves around the pair.
\nData Collector - it is the only component for which the logical diagram differs from the physical one, albeit by\ndesign. This stems from the fact that the Collector has been meant as the main extension point that other teams,\npossibly using disparate stacks, can easily plug into when adding new features. By all means, a Collector, as previously\nstated, is any component that fetches data from an arbitrary source, transforms it into Resources and/or Readings and\npushes it to the Data Sink SNS topic. In consequence, when you want to start collecting a new type of data or start collecting it from a different source, you\nhave two options - either add your implementation in one of already existing collector services, or write your own new\none, and the only requirement we superimpose on you is that when you publish to the Data Sink, you follow the format we\nhave specified. At the time this post is being written, we have 3 distinct Data Collector implementations: Main Collector - a general-purpose collector, written in Java and providing data for cross-platform features,\ne.g.: service ownership incidents Docker images used and their vulnerabilities service health and deployments currently running instances code insights (quality, PRs, contributors etc) Java Collector - yet another service written in Java, as the name suggests it collects data strictly related to\nservices written in Java, e.g.: runtime information like JVM and GC being used selective list of JVM parameters passed to the instances egress dependencies for Java services (in this context, the upstream services) PHP/Golang collector - the first contribution coming from outside the original App Cockpit team, written in\nGolang, it is going to provision data for PHP and Golang services, e.g.: egress dependencies vulnerabilities possibly many more to come With a picture being worth a thousand words, let’s have a look at a diagram that gathers all of this in one place: Obviously, the actual process of data collection, as it is currently implemented, is a whole story in and of itself,\ndeserving yet another dedicated blog post, especially since we have elected to make all the collectors stateless, which\nhad its numerous consequences and challenges so keep your eye out for the next entry which is bound to come shortly. Closing words and … we will be back Having said, or rather written all the above, we sincerely hope that we have managed to pique your interest. Not only in\nour solution but also in actually getting “concerned” about your ecosystem, because it does not necessarily have to be\nboring, especially when you have the right tools to make your life easier. On top of that, you can have your fun\nbuilding the actual tools, deftly combining business with pleasure for a change of the usual pace. As hinted numerous times, this is only the preliminary article, meant to lay the groundwork for future additions\nexploring individual pieces in more depth… We are already looking forward to writing those, so keep your eye out!\n
\n\n
\n\n
\n\n
\n\n
\n