Being a hero is nice, isn’t it? You work hard, single-handedly save the day, and your teammates are eternally grateful to you. However, such behavior is, in fact, highly problematic.

In Google, we have an internal document, quite often quoted, that says to let things fail if the workload to keep them alive is too high. From Google, many good practices have been exported and adopted in the IT world, and I believe this could be very useful to many teams.

Illustration by unDraw.

The story of how “heroism is bad” is not new. The internal Google document has actually been a presentation at SRE TechCon 2015. And, other companies have a similar take. But let’s start from the principle.

What is the significance of heroism?

When we talk about a hero for the team, we mean a single person taking on a systemic problem to compensate for it. No matter the cost, how many hours are needed, or the consequences. Of course, thanks to the hero, the crisis due to the systemic issue is momentarily avoided.

And, heroes get immediate reward! They have completed a gigantic task, they get praised from teammates, and they feel crucial: after all, without them, everything would have failed.

Why being a hero is bad

Of course, we are talking about behavior in the working environment, not of heroic acts to actually save lives. Heroic behavior, although quite encouraged in many realities, at the end of the day, brings more damage than benefits. It is bad for the individual: takes a toll on them, brings to burnout, creates unsatisfiable expectations (you did once, why don’t you do it again?), and puts a lot of pressure on them.

It is also bad for the team: it creates incorrect expectations from the team members, and it leads to inaction because the heroes will pick up anything left behind. Not only that, but it damages the morals because it makes it feel like everything has been solved by the hero, and everybody else has no purpose.

And it is bad for systems and processes: they don’t improve because teams don’t realize they are broken, given that at the end of the day, the heroes make everything work. No systemic fixes are carried on, ‘cause such problems are hidden due to the heroes efforts.

We work in teams for a reason: the idea that one person has the solution to everything is ridiculous.

What to do instead?

On YouTube, there is an astonishing TED talk by Lorna Davis, “A guide to collaborative leadership”. If you have 14 minutes of spare time today, you should really use them to watch it.

There is a key concept in the video: radical interdependence.

What’s radical interdependence?

Radical interdependence is just a fancy way to say that we need each other. In a complex, ever evolving, interconnected world, a single individual cannot change the world on its own.

And a team can delivery things that could be impossible to deliver by an individual alone.

However, quoting from the video:

Interdependence is harder than being a hero. It requires us to be open, transparent, and vulnerable. And that’s not what traditional leaders have been trained to do.

Let things fail

It’s hard, but nowadays, every time I feel the urge to step in to save the day, I stop a minute, and ask myself: “Can’t I sit down with the team and work this out?”. And every so often, this requires things to fail: but that’s okay.

It is imperative that a good process for postmortems is in place, and that such process is blameless.

When things fail, we sit down all together, and we find out what went wrong, and what we can do better the next time. This allows improvements to the system, to the process, and at the end things are remarkably better than what they would have been if they hadn’t failed.

It’s a complex, difficult approach, but it provides significant improvements. Have you ever tried it, or plan to do so? Can you think of an occasion where it would have been useful? Let me know in the comments!

If you have any question, other than here in the comments you can always reach me by email at hello@rpadovani.com.

Ciao,
R.

DS5QDR Heonmin Lee  ha creato (da diverso tempo) questa applicazione grafica riferendosi al precedente programma python pyUC, che è un semplice programma front-end GUI (con interfaccia grafica), per gestire l'accesso alle reti digitali dei Radioamatori, direttamente dal proprio PC.

Pochi giorni fa il Ministero dell'Industria e del Made in Italy (MIMIT) ha emanato il nuovo Decreto direttoriale che determina le modalità di svolgimento della sessione di esame per l'ottenimento della Patente di Radioamatore.

Da qualche settimana è uscito l'aggiornamento di sicurezza Windows per sistemi operativi 22H2 a 64 bit, che ho installato regolarmente senza alcun problema.
Di per se non ne avevo dato alcun peso, continuando ad utilizzare normalmente il mio PC Desktop, fino a quando non ho dovuto mettere mano alla programmazione di alcune radio di amici OM.

Continuando a sperimentare con i miei hotspot pi-star, realizzati su Raspberry Pi di varie versioni, nei giorni scorsi mi è stato chiesto come gestire velocemente la connessione iniziale all'AP (Access Point) di un nuovo hotspot pi-star, utilizzando la connessione WFi via PC.

Il network BrandMeister ha recentemente implementato il protocollo digitale radioamatoriale YSF Direct, ovvero la possibilità sui master server che hanno attiva questa funzionalità di collegare direttamente un hotspot/ripetitore C4FM in modalità YSF al master senza necessità di “room” intermedie come da prassi.

Negli ultimi fine settimana (durante il mio poco tempo libero...) ho effettuato diverse prove di configurazione sul mio server Rpi 3 B+ dove ho già installato alcuni nodi ASL. Lo scopo delle mie modifiche è stato quello di aggiungere ed abilitare un nodo radio aggiuntivo, oltre a quelli già presenti e funzionanti

The Certified Kubernetes Administrator (CKA) exam evaluates your ability to operate a Kubernetes (K8s) cluster and your knowledge of how to run jobs over the cluster.

I recently passed the exam, so I would like to share some tips and tricks, and how I prepared for it.

Illustration by unDraw.

Preparation

Before taking the exam, I already had extensive experience using K8s, however always through managed services. I had to learn everything about how a K8s cluster is actually run.

I am grateful to my employer, Google, for providing me with the time to study and covering all expenses.

I used the course hosted over A Cloud Guru. It is up-to-date, and covers everything needed for the CKA. The course is very hands-on, so if you sign up for it, I recommend doing the laboratories (they provide the environment) because the exam is a long series of tasks similar to what you would be asked to do during the lab sections of this course.

Furthermore, you should learn how to navigate the K8s doc. You can use it during the exam, so there is no point in learning every single field of every single resource. Do you need to create a pod? Go to the documentation for it, copy the basis example, and customize it.

Simulation

When you register for the CKA certification exam on the Linux Foundation site, you will receive access to a simulated exam hosted by killer.sh. The simulation is significantly more difficult and longer than the actual exam: I attempted a simulation two days before the exam, and I achieved 55%. On the actual exam, 91%.

I highly recommend taking the simulation, not only to see how prepared you are, but to familiarize yourself with the exam environment.

The exam

The exam is 2 hours long, with a variable number of tasks. Points are also assigned based on how many tasks of a question you answered. You need a score of 66% to clear the exam.

During the exam you can access K8s documentation, use this power to your advantage. The environment in which the exam will run is based on XFCE. You will have a browser, a note pad, and a terminal you can use.

Being familiar with vi can help you work more quickly, especially if you spend a lot of time in there, and you just use the browser to read the documentation.

You can use the note pad to keep track of which questions you skipped, or you want to revise later.

During your exam, you will work on multiple K8s clusters. Each question is about one cluster, and at the beginning of the question, it says which context you should use. Don’t skip to answer the question without having switched context!

Requirements

To attend the exam, you will have to download a dedicated browser, called PSI Safe Browser. You cannot download this in advance, it will be unlocked for you only 30 minutes before the beginning of the exam.

Theoretically, it is compatible with Windows, macOS, and Linux: however, many users have problems under Linux. Be aware that there could be issues, and have a macOS or Windows machine available, if possible, as a backup plan.

This browser will make sure that you are not running any background application, and that you don’t have multiple displays (only one monitor, sorry!).

After you installed the browser, you will have to identify yourself with some official document. You will have to show that you are alone in the room, that your desk is clean, that there is nothing under your desk, and so on. You will have to show your smartphone, and then show that you place it in some unreachable place. The whole check-in procedure takes time, and it is error-prone (I had to show the whole room twice, then the browser crashed, and I had to start from scratch again).

The onboarding procedure is the most stressful part of the exam: read all the requirements on the Linux Foundation Website, try to clear your desk from any unrelated thing, and start the procedure as soon as possible. In my case, it took almost 50 minutes.

Their browser is not a perfect piece of software: in my case, the virtual environment inside this browser had a resolution of 800x600, making it impossible to have two windows on a side-by-side. However, you spend the huge majority of your time in the terminal, and sometimes on the browser to copy-paste snippets from the browser.

Tips

• Keep a Windows or macOS machine nearby, if Linux doesn’t work for you;
• Answering a question partially is better than not answering at all;
• Always double-check the K8s context! First thing you must do for each question, it is switching the context of your kubectl according to instructions;
• Create files for each resource you create, so you can go back and adjust stuff. Moreover, if you have time at the end of the exam, makes way easier to check what you have done;
• Remember to have a valid ID document with you for the exam check-in;

Conclusion

Time constraints and the unfamiliar environment will be your greatest challenges: however, if you have used K8s in production before, you should be able to clear the exam without any major difficulty. Spending sometime training before is strongly suggested, no matter your level of expertise, just to understand the format of the exam.

Good luck, and reach out if you have any question, here in the comments or by email at hello@rpadovani.com.

Ciao,
R.

Qui trovate le informazioni con le quali installerete la nuova versione 7.2+ di Supermon (versione ASL), denominata Fresh-Install.  Questa installazione comprende tutti i file di configurazione standard di Supermon.

To create a Kubernetes deployment, we must specify the matchLabels field, even though its value must match the one we specify in the template. But why? Cannot Kubernetes be smart enough to figure it out without us being explicit?

Illustration by unDraw+.

Did you know? K8s is short for Kubernetes ‘cause there are 8 letters between K and S.

A Kubernetes (K8s) Deployment provides a way to define how many replicas of a Pod K8s should aim to keep alive. I’m especially bothered by the Deployment spec’s requirement that we must specify a label selector for pods, and that label selector must match the same labels we have defined in the template. Why can’t we just define them once? Why can’t K8s infer them on its own? As I will explain, there is actually a good reason. However, to understand it, you would have to go down a rabbit hole to figure it out.

A deployment specification

Firstly, let’s take a look at a simple deployment specification for K8s:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx # Why can't K8s figure it out on its own?
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest

This is a basic deployment, taken from the official documentation, and here we can already see that we need to fill the matchLabels field.

What happens if we drop the “selector” field completely?

➜ kubectl apply -f nginx-deployment.yaml
The Deployment "nginx-deployment" is invalid:
* spec.selector: Required value

Okay, so we need to specify a selector. Can it be different from the “label” field in the “template”? I will try with:

matchLabels:
      app: nginx-different

➜ kubectl apply -f nginx-deployment.yaml
The Deployment "nginx-deployment" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"nginx"}: `selector` does not match template `labels`

There are usually good reasons behind what it seems a not well-thought implementation, and this is true here as well, as we’ll see.

As expected, K8s doesn’t like it, it must match the template. So, we must fill a field with a well-defined value. It really seems that a computer could do that for us, why do we have to specify it manually? It drives me crazy having to do something a computer could do without any problem. Or could it?

Behind a deployment

Probably, you may never need to manipulate ReplicaSet objects: use a Deployment instead, and define your application in the spec section.

How does a deployment work? Behind the curtains, when you create a new deployment, K8s creates two different objects: a Pod definition, using as its specification what is available in the “template” field of the Deployment, and a ReplicaSet. You can easily verify this using kubectl to retrieve pods and replica sets after you have created a deployment.

A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.

A ReplicaSet needs a selector that specifies how to identify Pods it can acquire and manage: however, this doesn’t explain why we must specify it, and why K8s cannot do it on its own. In the end, a Deployment is a high-level construct that should hide ReplicaSet quirks: such details shouldn’t concern us, and the Deployment should take care of them on its own.

Digging deeper

Understanding how a Deployment works doesn’t help us find the reason of this particular behavior. Given that Googling doesn’t seem to bring any interesting result on this particular topic, it’s time to go to the source (literally): luckily, K8s is open source, so we can check its history on GitHub.

Going back in time, we find out that, actually, K8s used to infer it the matchLabels field! The behavior was removed with apps/v1beta2 (released with Kubernetes 1.8), through Pull Request #50164. Such pull request links to issue #50339, that, however, has a very brief description, and lacks the reasoning behind such a choice.

The linked issues are rich of technical details, and they have many comments. If you want to understand exactly how kubectl apply works, take a look!

Luckily, other issues provide way more context, as #26202: it turns out, the main problem with defaulting is when in subsequent updates to the resource, labels are mutated: the patch operation is somehow fickle, and apply breaks when you update a label that was used as default.

Many other concerns have been described by Brian Grant in deep in the issue #15894.

Basically, assuming a default value, creates many questions and concerns: what’s the difference between explicitly setting a label as null, and leaving it empty? How to manage all the cases where users left the default, and now they want to update the resource to manage themselves the label (or the other way around)?

Conclusion

Given that in K8s everything intend to be declarative, developers have chosen that explicit is better than implicit, especially for corner cases: specifying things explicitly allows a more robust validation on creation and update time, and remove some possible bugs that existed due to uncertainties caused by lack of clarity.

Shortly after dropping the defaulting behavior, developers also made the labels immutable, to make sure behaviors were well-defined. Maybe in the future labels will be mutable again, but to have that working, somebody needs to design a well-though document explaining how to manage all the possible edge cases that could happen when a controller is updated.

I hope you found this deep dive into the question interesting. I spent some time on it, since I was very curious, and I hope that the next person with the same question can find this article and get an answer quicker than what it took me.

If you have any question, or feedback, please leave a comment below, or write me an email at hello@rpadovani.com.

Ciao,
R.

Da qualche giorno, e anche ieri sera, parlandone assieme al mio mentore Mauro IU0NDT, lui stesso mi ha consigliato di installare la versione 7 del node manager Supermon (la nuova versione 7.1 + l'ho trovata bella e pronta), che può coesistere come servizio web assieme all'altro gestore  nodi precedente.

Pochi giorni fa è stata distribuita la versione Stabile 2.0.0 dell'App Android DVSwitch Mobile. In questi giorni ho fatto diverse prove utilizzando questa App Android sui miei nodi ASL e collegandone altri anche in modalità WT (Web Trasceiver).

Da qualche giorno è uscita la nuova versione del firmware in oggetto, nella quale sono state aggiunte diverse cose, che potrete leggere consultando il documento presente nel pacchetto della distribuzione firmware.

Helm is a remarkable piece of technology to manage your Kubernetes deployments, and used along Terraform is perfect for deploying following the GitOps strategy.

Illustration by unDraw+.

What’s GitOps? Great question! As this helpful, introductory article summarize, it is Infrastructure as Code, plus Merge Requests, plus Continuous Integration. Follow the link to explore further the concept.

However, Helm has a limitation: it doesn’t manage the lifecycle of Custom Resource Definitions (CRDs), meaning it will only install the CRD during the first installation of a chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed.

This can be a huge problem for a GitOps approach: having to update CRDs manually isn’t a great strategy, and makes it very hard to keep in sync with deployments and rollbacks.

For this very reason, I created a small Terraform module that will read from some online manifests of CRDs, and apply them. When parametrizing the version of the chart, it is simple to keep Helm Charts and CRDs in sync, without having to do anything manually.

Example

Karpenter is an incredible open-source Kubernetes node provisioner built by AWS. If you haven’t tried it yet, take some minutes to read about it.

Let’s use Karpenter as an example on how to use the module. We want to deploy the chart with the Helm provider, and we use this new Terraform module to manage its CRDs as well:

resource "helm_release" "karpenter" {
  name            = "karpenter"
  namespace       = "karpenter"
  repository      = "https://charts.karpenter.sh"
  chart           = "karpenter"
  version         = var.chart_version

  // ... All the other parameters necessary, skipping them here ...
}

module "karpenter-crds" {
  source  = "rpadovani/helm-crds/kubectl"
  version = "0.1.0"
  
  crds_urls = [
    "https://raw.githubusercontent.com/aws/karpenter/v${var.chart_version}/charts/karpenter/crds/karpenter.sh_provisioners.yaml",
    "https://raw.githubusercontent.com/aws/karpenter/v${var.chart_version}/charts/karpenter/crds/karpenter.k8s.aws_awsnodetemplates.yaml"
  ]
}

As you can see, we parametrize the version of the chart, so we can be sure to have the same version for CRDs as the Helm chart. Behind the curtains, Terraform will download the raw file, and apply it with kubectl. Of course, the operator running Terraform needs to have enough permissions to launch such scripts, so you need to configure the kubectl provider.

The URLs must point to just the Kubernetes manifests, and this is why we use the raw version of the GitHub URL.

The source code of the module is available on GitHub, so you are welcome to chime in and open any issue: I will do my best to address problems and implement suggestions.

Conclusion

I use this module in production, and I am very satisfied with it: it brings under GitOps the last part I missed: the CRDs. Now, my only task when I install a new chart is finding all the CRDs, and build a URL that contains the chart version. Terraform will take care of the rest.

I hope this module can be useful to you as it is to me. If you have any question, or feedback, or if you would like some help, please leave a comment below, or write me an email at hello@rpadovani.com.

Ciao,
R.

Contributing to any open-source project is a great way to spend a few hours each month. I started more than 10 years ago, and it has ultimately shaped my career in ways I couldn’t have imagined!

The new GitLab logo, just announced on the 27th April 2022.

Nowadays, my contributions focus mostly on GitLab, so you will see many references to it in this blog post, but the content is quite generalizable; I would like to share my experience to highlight why you should consider contributing to an open-source project.

Writing blog posts, tweeting, and helping foster the community are nice ways to contribute to a project ;-)

And contributing doesn’t mean only coding: there are countless ways to help an open-source project: translating it to different languages, reporting issues, designing new features, writing the documentation, offering support on forums and StackOverflow, and so on.

Before deep diving in this wall of text, be aware there are mainly three parts in this blog post, after this introduction: a context section, where I describe my personal experience with open source, and what does it mean to me. Then, a nice list of reasons to contribute to any project. In closing, there are some tips on how to start contributing, both in general, and something specific to GitLab.

Context

Ten years ago, I was fresh out of high school, without (almost) any knowledge about IT: however, I found out that I had a massive passion for it, so I enrolled in a Computer Engineering university (boring!), and I started contributing to Ubuntu (cool!). I began with the Italian Local Community, and I soon moved to Ubuntu Touch.

I often considered rewriting that old article: however I have a strange attachment to it as it is, with all that English mistakes. One of the first blog posts I have written, and it was really well received! We all know how it ended, but it still has been a fantastic ride, with a lot of great moments: just take a look at the archive of this blog, and you can see the passion and the enthusiasm I had. I was so enthusiast, that I wrote a similar blog post to this one! I think it highlights really well some considerable differences 10 years make.

Back then, I wasn’t working, just studying, so I had a lot of spare time. My English was way worse. I was at the beginning of my journey in the computer world, and Ubuntu has ultimately shaped a big part of it. My knowledge was very limited, and I never worked before. Contributing to Ubuntu gave me a glimpse of real world, I met outstanding engineers who taught me a lot, and boosted my CV, helping me to land my first job.

Advocacy, as in this blog post, is a great way to contribute! You spread awareness, and this helps to find new contributors, and maybe inspiring some young student to try out! Since then, I completed a master’s degree in C.S., worked in different companies in three different countries, and became a professional. Nowadays, my contributions to open source are more sporadic (adulthood, yay), but given how much it meant to me, I am still a big fan, and I try to contribute when I can, and how I can.

Why contributing

Friends

During my years contributing to open-source software, I’ve met countless incredible people, with some of whom I’ve become friend. In the old blog post I mentioned David: in the last 9 years we stayed in touch, met in different occasions in different cities: last time was as recent as last summer. Back at the time, he was a Manager in the Ubuntu Community Team at Canonical, and then, he became Director of Community Relations at GitLab. Small world!

The Ubuntu Touch Community Team in Malta, in 2014. It has been an incredible week, sponsored by Canonical!

One interesting thing is people contribute to open-source projects from their homes, all around the world: when I travel, I usually know somebody living in my destination city, so I’ve always at least one night booked for a beer with somebody I’ve met only online; it’s a pleasure to speak with people from different backgrounds, and having a glimpse in their life, all united by one common passion.

Fun

Having fun is important! You cannot spend your leisure time getting bored or annoyed: contributing to open source is fun ‘cause you pick the problems you would like to work on, and you don’t need all that bureaucracy and meetings that is often needed in your daily job. You can be challenged, and feeling useful, and improving a product, without any manager on your shoulder, and with your pace.

Being up-to-date on how things evolve

For example, the GitLab Handbook is a precious collection of resources, ideas, and methodologies on how to run a 1000 people company in a transparent, full remote, way. It’s a great reading, with a lot of wisdom.

Contributing to a project typically gives you an idea on how teams behind it work, which technologies they use, and which methodologies. Many open-source projects use bleeding-edge technologies, or draw a path. Being in contact with new ideas is a great way to know where the industry is headed, and what are the latest news: it is especially true if you hang out in the channels where the community meets, being them Discord, forums, or IRC (well, IRC is not really bleeding-edge, but it is fun).

Learning

When contributing in an area that doesn’t match your expertise, you always learn something new: reviews are usually precise and on point, and projects of a remarkable size commonly have a coaching team that help you to start contributing, and guide you on how to land your first patches.

In GitLab, if you need a help in merging your code, there are the Merge Request Coaches! And for any type of help, you can always join Gitter, or ask on the forum, or write to the dedicated email address.

Feel also free to ping me directly if you want some general guidance!

Giving back

I work as a Platform Engineer. My job is built on an incredible amount of open-source libraries, amazing FOSS services, and I basically have just to glue together different pieces. When I find some rough edge that could be improved, I try to do so.

Nowadays, I find crucial having well-maintained documentation, so after I have achieved something complex, I usually go back and try to improve the documentation, where lacking. It is my tiny way to say thanks, and giving back to a world that really has shaped my career.

This is also what mostly of my blogs posts are about: after having completed something on which I spent fatigue on, I find it nice being able to share such information. Every so often, I find myself years later following my guide, and I really also appreciate when other people find the content useful.

Swag

Who doesn’t like swag? :-) Numerous projects have delightful swags, starting from stickers, that they like to share with the whole community. Of course, it shouldn’t be your main driver, ‘cause you will soon notice that it is ultimately not worth the amount of time you spend contributing, but it is charming to have GitLab socks!

A GitLab branded mechanical keyboard, courtesy of the GitLab's security team! This very article has been typed with it!

Tips

I hope I inspired you to contribute to some open-source project (maybe GitLab!). Now, let’s talk about some small tricks on how to begin easily.

Find something you are passionate about

You must find a project you are passionate about, and that you use frequently. Looking forward to a release, knowing that your contributions will be included, it is a wonderful satisfaction, and can really push you to do more.

Moreover, if you already know the project you want to contribute to, you probably know already the biggest pain points, and where the project needs some contributions.

Start small and easy

You don’t need to do gigantic contributions to begin. Find something tiny, so you can get familiar with the project workflows, and how contributions are received.

Launchpad and bazaar instead of GitLab and git — down the memory lane! My journey with Ubuntu started correcting a typo in a README, and here I am, years later, having contributed to dozens of projects, and having a career in the C.S. field. Back then, I really had no idea of what my future would have held.

For GitLab, you can take a look at the issues marked as “good for new contributors”. They are designed to be addressed quickly, and onboard new people in the community. In this way, you don’t have to focus on the difficulties of the task at hand, but you can easily explore how the community works.

Writing issues is a good start

Writing high-quality issues is a great way to start contributing: maintainers of a project are not always aware of how the software is used, and cannot be aware of all the issues. If you know that something could be improved, write it down: spend some time explicating what happens, what you expect, how to reproduce the problem, and maybe suggest some solutions as well! Perhaps, the first issue you write down could be the very first issue you resolve.

Not much time required!

Contributing to a project doesn’t require necessarily a lot of time. When I was younger, I definitely dedicated way more time to open-source projects, implementing gigantic features. Nowadays, I don’t do that anymore (life is much more than computers), but I like to think that my contributions are still useful. Still, I don’t spend more than a couple of hours a month, based on my schedule, and how much is raining (yep, in winter I definitely contribute more than in summer).

GitLab is super easy

Do you use GitLab? Then you should undoubtedly try to contribute to it. It is easy, it is fun, and there are many ways. Take a look at this guide, hang out on Gitter, and see you around. ;-)

Next week (9th-13th May 2022) there is also a GitLab Hackathon! It is a real fun and easy way to start contributing: many people are available to help you, there are video sessions talking about contributing, and just doing a small contribution you will receive a pretty prize.

And if I was able to do it with my few contributions, you can as well! And in time, if you are consistent in your contributions, you can become a GitLab Hero! How cool is that?

I really hope this wall of text made you consider contributing to an open-source project. If you have any question, or feedback, or if you would like some help, please leave a comment below, or write me an email at hello@rpadovani.com.

Ciao,
R.

Rust is all hot these days, and it is indeed a nice language to work with. In this blog post, I take a look at a small challenge: how to host private crates in the form of Git repositories, making them easily available both to developers and CI/CD systems.

Ferris the crab, unofficial mascot for Rust.

A Rust crate can be hosted in different places: on a public registry on crates.io, but also in a private Git repo hosted somewhere. In this latter case, there are some challenges on how to make the crate easily accessible to both engineers and CI/CD systems.

Developers usually authenticate through SSH keys: given humans are terrible at remembering long passwords, using SSH keys allow us to do not memorize credentials, and having authentication methods that just works. The security is quite high as well: each device has a unique key, and if a device gets compromised, deactivating the related key solves the problem.

On the other hand, it is better avoiding using SSH keys for CI/CD systems: such systems are highly volatile, with dozens, if not hundreds, instances that get created and destroyed hourly. For them, a short-living token is a way better choice. Creating and revoking SSH keys on the fly can be tedious and error-prone.

This arises a challenge with Rust crates: if hosted on a private repository, the dependency can be reachable through SSH, such as

[dependencies]
my-secret-crate = { git = "ssh://git@gitlab.com/rpadovani/my-secret-crate.git", branch = "main" }

or through HTTPS, such as

[dependencies]
my-secret-crate = { git = "https://gitlab.com/rpadovani/my-secret-crate", branch = "main" }

In the future, I hope we don’t need to host private crates on Git repositories, GitLab should add a native implementation of a private registry for crates.

The former is really useful and simple for engineers: authentication is the same as always, so no need to worry about it. However, it is awful for CI/CD: now there is need to manage the lifecycle of SSH keys for automatic systems.

The latter is awful for engineers: they need a new additional authentication method, slowing them down, and of course there will be authentication problems. On the other hand, it is great for automatic systems.

How to conciliate the two worlds?

Well, let’s use them both! In the Cargo.toml file, use the SSH protocol, so developers can simply clone the main repo, and they will be able to clone the dependencies without further hassle.

Then, configure the CI/CD system to clone every dependency through HTTPS, thanks to a neat feature of Git itself: insteadOf.

From the Git-SCM website:

url.<base>.insteadOf:
Any URL that starts with this value will be rewritten to start, instead, with . In cases where some site serves a large number of repositories, and serves them with multiple access methods, and some users need to use different access methods, this feature allows people to specify any of the equivalent URLs and have Git automatically rewrite the URL to the best alternative for the particular user, even for a never-before-seen repository on the site. When more than one insteadOf strings match a given URL, the longest match is used.

Basically, it allows rewriting part of the URL automatically. In this way, it is easy to change the protocol used: developers will be happy, and the security team won’t have to scream about long-lived credentials on CI/CD systems.

Do you need an introduction to GitLab CI/CD? I’ve written something about it!

An implementation example using GitLab, but it can be done on any CI/CD system:

my-job:
  before_script:
    - git config --global credential.helper store
    - echo "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com" > ~/.git-credentials
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf ssh://git@gitlab.com
  script:
    - cargo build

The CI_JOB_TOKEN is a unique token valid only for the duration of the GitLab pipeline. In this way, also if a machine got compromised, or logs leaked, the code is still sound and safe.

What do you think about Rust? If you use it, have you integrated it with your CI/CD systems? Share your thoughts in the comments below, or drop me an email at riccardo@rpadovani.com.

Ciao,
R.

AWS EKS is a remarkable product: it manages Kubernetes for you, letting you focussing on creating and deploying applications. However, if you want to manage permissions accordingly to the shared responsibility model, you are in for some wild rides.

Image courtesy of unDraw.

The shared responsibility model

First, what’s the shared responsibility model? Well, to design a well-architected application, AWS suggests following six pillars. Among these six pillars, one is security. Security includes sharing responsibility between AWS and the customer. In particular, and I quote,

Customers are responsible for managing their data (including encryption options), classifying their assets, and using IAM tools to apply the appropriate permissions.

Beautiful, isn’t it? AWS gives us a powerful tool, IAM, to manage permissions; we have to configure things in the best way, and AWS gives us the way to do so. Or does it? Let’s take a look together.

Our goal

I would say the goal is simple, but since we are talking about Kubernetes, things cannot be just simple.

Our goal is quite straightforward: setting up a Kubernetes cluster for our developers. Given that AWS offers AWS EKS, a managed Kubernetes service, we only need to configure it properly, and we are done. Of course, we will follow best practices to do so.

A proper setup

Infrastructure as code is out of scope for this post, but if you have never heard about it before, I strongly suggest taking a look into it.

Of course, we don’t use the AWS console to manually configure stuff, but Infrastructure as Code: basically, we will write some code that will call the AWS APIs on our behalf to set up AWS EKS and everything correlated. In this way, we can have a reproducible setup that we could deploy in multiple environments, and countless other advantages.

Moreover, we want to avoid launching scripts that interact with our infrastructure from our PC: we prefer not to have permissions to destroy important stuff! Separation of concerns is a fundamental, and we want to write code without worrying about having consequences on the real world. All our code should be vetted from somebody else through a merge request, and after being approved and merged to our main branch, a runner will pick it up and apply the changes.

A runner is any CI/CD that will execute the code on your behalf. In my case, it is a GitLab runner, but it could be any continuous integration system.

We are at the core of the problem: our runner should follow the principle of least privilege: it should be able to do only what it needs to do, and nothing more. This is why we will create a IAM role only for it, with only the permissions to manage our EKS cluster and everything in it, but nothing more.

I would have a massive rant about how bad is the AWS documentation for IAM in general, not only for EKS, but I will leave it to some other day.

The first part of creating a role with minimum privileges is, well, understanding what minimum means in our case. A starting point is the AWS documentation: unfortunately, it is always a bad starting point concerning IAM permissions, ‘cause it is always too generous in allowing permissions.

The “minimum” permission accordingly to AWS

According to the guide, the minimum permissions necessary for managing a cluster is being able to do any action on any EKS resource. A bit farfetched, isn’t it?

Okay, hardening this will be fun, but hey, do not let bad documentations get in the way of a proper security posture.

You know what will get in the way? Bugs! A ton of bugs, with absolutely useless error messages.

I started limiting access to only the namespace of the EKS cluster I wanted to create. I ingenuously thought that we could simply limit access to the resources belonging to the cluster. But, oh boy, I was mistaken!

Looking at the documentation for IAM resources and actions, I created this policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "eks:ListClusters",
                "eks:DescribeAddonVersions",
                "eks:CreateCluster"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "eks:*",
            "Resource": [
                "arn:aws:eks:eu-central-1:123412341234:addon/my-cluster/*/*",
                "arn:aws:eks:eu-central-1:123412341234:fargateprofile/my-cluster/*/*",
                "arn:aws:eks:eu-central-1:123412341234:identityproviderconfig/my-cluster/*/*/*",
                "arn:aws:eks:eu-central-1:123412341234:nodegroup/my-cluster/*/*",
                "arn:aws:eks:eu-central-1:123412341234:cluster/my-cluster"
            ]
        }
    ]
}

Unfortunately, if a role with these permissions try to create a cluster, this error message appears:

Error: error creating EKS Add-On (my-cluster:kube-proxy): AccessDeniedException: User: arn:aws:sts::123412341234:assumed-role/<role>/<iam-user> is not authorized to perform: eks:TagResource on resource: arn:aws:eks:eu-central-1:123412341234:/createaddon

I have to say that at least the error message gives you a hint: the /createddon action is not scoped to the cluster.

After fighting with different polices for a while, I asked DuckDuckGo for a help, and indeed somebody reported this problem to AWS before, in this GitHub issue.

What the issue basically says is that if we want to give an IAM role permission to manage an add-on inside a cluster, we must give it permissions over all the EKS add-ons in our AWS account.

This of course breaks the AWS shared responsibility principle, ‘cause they don’t give us the tools to upheld our part of the deal. This is why it is a real and urgent issue, as they also mention in the ticket:

Can’t share a timeline in this forum, but it’s a high priority item.

And indeed it is so high priority, that it has been reported the 3rd December 2020, and today, more than one year later, the issue is still there.

To add insult to the injury, you have to write the right policy manually because if you use the IAM interface to select “Any resource” for the add-ons as in the screenshot below, it will generate the wrong policy! If you check carefully, the generated resource name is arn:aws:eks:eu-central-1:123412341234:addon/*/*/*, which of course doesn’t match the ARN expected by AWS EKS. Basically, also if you are far too permissive, and you use the tools that AWS provides you, you still will have some broken policy.

Do you have some horror story about IAM yourself? I have a lot of them, and I am thinking about a more general post. What do you think? Share your thoughts in the comments below, or drop me an email at riccardo@rpadovani.com.

Ciao,
R.

Terraform is a powerful tool. However, it has some limitations: since it uses AWS APIs, it doesn’t have a native way to check if an EC2 instance has completed to run cloud-init before marking it as ready. A possible workaround is asking Terraform to SSH on the instance, and wait until it is able to perform a connection before marking the instance as ready.

Terraform logo, courtesy of HashiCorp.

I find using SSH in Terraform quite problematic: you need to distribute a private SSH key to anybody that will launch the Terraform script, including your CI/CD system. This is a no-go for me: it adds the complexity to manage SSH keys, including their rotation. There is a huge issue on the Terraform repo on GitHub about this functionality, and the most voted solution is indeed connecting via SSH to run a check:

provisioner "remote-exec" {
  inline = [
    "cloud-init status --wait"
  ]
}

AWS Systems Manager Run Command

The idea of using cloud-init status --wait is indeed quite good. The only problem is how do we ask Terraform to run such command. Luckily for us, AWS has a service, AWS SSM Run Command that allow us to run commands on an EC2 instance through AWS APIs! In this way, our CI/CD system needs only an appropriate IAM role, and a way to invoke AWS APIs. I use the AWS CLI in the examples below, but you can adapt them to any language you prefer.

Prerequisites

If you don’t know AWS SSM yet, go and take a look to their introductory guide. There are some prerequisites to use AWS SSM Run Command: we need to have AWS SSM Agent installed on our instance. It is preinstalled on Amazon Linux 2 and Ubuntu 16.04, 18.04, and 20.04. For any other OS, we need to install it manually: it is supported on Linux, macOS, and Windows.

The user or the role that executes the Terraform code needs to be able to create, update, and read AWS SSM Documents, and run SSM commands. A possible policy could be look like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1629387563127",
      "Action": [
        "ssm:CreateDocument",
        "ssm:DeleteDocument",
        "ssm:DescribeDocument",
        "ssm:DescribeDocumentParameters",
        "ssm:DescribeDocumentPermission",
        "ssm:GetDocument",
        "ssm:ListDocuments",
        "ssm:SendCommand",
        "ssm:UpdateDocument",
        "ssm:UpdateDocumentDefaultVersion",
        "ssm:UpdateDocumentMetadata"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

If we already know the name of the documents, or the instances where we want to run the commands, it is better to lock down the policy specifying the resources, accordingly to the principle of least privilege.

Last but not least, we need to have the AWS CLI installed on the system that will execute Terraform.

The Terraform code

After having set up the prerequisites as above, we need two different Terraform resources. The first will create the AWS SSM Document with the command we want to execute on the instance. The second one will execute such command while provisioning the EC2 instance.

The AWS SSM Document code will look like this:

resource "aws_ssm_document" "cloud_init_wait" {
  name = "cloud-init-wait"
  document_type = "Command"
  document_format = "YAML"
  content = <<-DOC
    schemaVersion: '2.2'
    description: Wait for cloud init to finish
    mainSteps:
    - action: aws:runShellScript
      name: StopOnLinux
      precondition:
        StringEquals:
        - platformType
        - Linux
      inputs:
        runCommand:
        - cloud-init status --wait
    DOC
}

We can refer such document from within our EC2 instance resource, with a local provisioner:

resource "aws_instance" "example" {
  ami           = "my-ami"
  instance_type = "t3.micro"

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]

    command = <<-EOF
    set -Ee -o pipefail
    export AWS_DEFAULT_REGION=${data.aws_region.current.name}

    command_id=$(aws ssm send-command --document-name ${aws_ssm_document.cloud_init_wait.arn} --instance-ids ${self.id} --output text --query "Command.CommandId")
    if ! aws ssm wait command-executed --command-id $command_id --instance-id ${self.id}; then
      echo "Failed to start services on instance ${self.id}!";
      echo "stdout:";
      aws ssm get-command-invocation --command-id $command_id --instance-id ${self.id} --query StandardOutputContent;
      echo "stderr:";
      aws ssm get-command-invocation --command-id $command_id --instance-id ${self.id} --query StandardErrorContent;
      exit 1;
    fi;
    echo "Services started successfully on the new instance with id ${self.id}!"

    EOF
  }
}

From now on, Terraform will wait for cloud-init to complete before marking the instance ready.

Conclusion

AWS Session Manager, AWS Run Commands, and the others tools in the AWS Systems Manager family are quite powerful, and in my experience they are not widely use. I find them extremely useful: for example, they also allows connecting via SSH to the instances without having any port open, included the 22! Basically, they allow managing and running commands inside instances only through AWS APIs, with a lot of benefits, as they explain:

Session Manager provides secure and auditable instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys. Session Manager also allows you to comply with corporate policies that require controlled access to instances, strict security practices, and fully auditable logs with instance access details, while still providing end users with simple one-click cross-platform access to your managed instances.

DO you have any questions, feedback, critics, request for support? Leave a comment below, or drop me an email at riccardo@rpadovani.com.

Ciao,
R.

After years of blogging, I’ve finally chosen to add a comment system, including reactions, to this blog. I’ve done so to make it easier engaging with the four readers of my blabbering: of course, it took some time to choose the right comment provider, but finally, here we are!

A picture I took at the CCC 2015 - and the moderation policy for comments!

I’ve chosen, long ago, to do not put any client-side analytics on this website: while the nerd in me loves graph and numbers, I don’t think they are worthy of the loss of privacy and performance for the readers. However, I am curious to have feedback on the content I write, if some of them are useful, and how can I improve. In all these years, I’ve received different emails about some posts, and they are heart-warming. With comments, I hope to reduce the friction in communicating with me and having some meaningful interaction with you.

Enter hyvor.com

Looking for a comment system, I had three requirements in mind: being privacy-friendly, being performant, and being managed by somebody else.

A lot of comments system are “free”, because they live on advertisements, or tracking user activities, and more often a combination of both. However, since I want comments on my website, I find dishonest that you, the user, has to pay the price for it. So, I was looking for something that didn’t track users, and bases its business on dear old money. My website, my wish, my wallet.

I find performances important, and unfortunately, quite undervalued on the bloated Internet. I like having just some HTML, some CSS, and the minimum necessary of JavaScript, so whoever stumbles on these pages don’t waste CPU and time waiting for some rendering. Being a static website, there isn’t a server side, so I cannot put comments there. I had to find a really light JavaScript based comment system.

Given these two prerequisites, you would say the answer is obvious: find a comment system that you can self-host! And you would be perfectly right - however, since I spend already 8 hours a day keeping stuff online, I really don’t want to have to care about performance and uptime in my free time - I definitely prefer going drinking a beer with a friend.

After some shopping around, I’ve chosen to go with Hyvor Talk, since it checks all three requirements above. I’ve read nice things about it, so let’s see how it goes! And if you don’t see comments at the end of the page, probably a privacy plugin for your browser is blocking it - up to you if you want to whitelist my website, or communicate with me in other ways ;-)

A nice plus of Hyvor is they also support reactions, so if you are in an hurry but want still to leave a quick feedback on the post, you can simply click a button. Fancy, isn’t it?

Moderation

Internet can be ugly sometime, and this is why I will keep a strict eye on the comments, and I will probably adjust moderation settings in the future, based on how things evolve - maybe no-one will comment, then no need for any strict moderation! The only rule I ask you to abide, and I’ve put as moderation policy, is: “Be excellent to each other”. I’ve read it at the CCC Camp 2015, and it sticks with me: as every short sentence, cannot capture all the nuances of human interaction, but I think it is a very solid starting point. If you have any concern or feedback you prefer to do not express in public, feel free to reach me through email. Otherwise, I hope to see you all in the comment section ;-)

Questions, comments, feedback, critics, suggestions on how to improve my English? Leave a comment below, or drop me an email at riccardo@rpadovani.com. Or, from today, leave a comment below!

Ciao,
R.

“Build smaller, faster, and more secure desktop applications with a web frontend” is the promise made by Tauri. And indeed, it is a great Electron replacement. But being in its first days (the beta has just been released!) a bit of documentation is still missing, and on the internet there aren’t many examples on how to write code.

Tauri is very light, as highlighted by these benchmarks.

Haven’t you started with Tauri yet? Go and read the intro doc! One thing that took me a bit of time to figure out, is how to read environment variables from the frontend of the application (JavaScript / Typescript or whichever framework you are using).

The Command

As usual, when you know the solution the problem seems easy. In this case, we just need an old trick: delegating!

In particular, we will use the Command API to spawn a sub-process (printenv) and reading the returned value. The code is quite straightforward, and it is synchronous:

import { Command } from "@tauri-apps/api/shell";

async readEnvVariable(variableName: string): Promise<string> {
    const commandResult = await new Command(
        "printenv",
        variableName
    ).execute();

    if (commandResult.code !== 0) {
      throw new Error(commandResult.stderr);
    }
    
    return commandResult.stdout;
}

The example is in Typescript, but can be easily transformed in standard JavaScript.

The Command API is quite powerful, so you can read its documentation to adapt the code to your needs.

Requirements

There are another couple of things you should consider: first, you need to have installed in your frontend environment the @tauri-apps/api package. You also need to enable your app to execute commands. Since Tauri puts a strong emphasis on the security (and rightly so!), your app is by default sandboxed. To being able to use the Command API, you need to enable the shell.execute API.

It should be as easy as setting tauri.allowlist.shell.execute to true in your tauri.json config file.

Tauri is very nice, and I really hope it will conquer the desktops and replace Electron, since it is lighter, faster, and safer!

Questions, comments, feedback, critics, suggestions on how to improve my English? Leave a comment below, or drop me an email at riccardo@rpadovani.com.

Ciao,
R.

Want to quit from a conversation with a bot provided through Dialogflow? You can use an Italian city, too.

Imagine that you go to the registry office.

The employee has to ask different questions, like your name, birthday, and so on.

He just asked the first one, you expect a lot more questions.

Second one 

BOT: “What’s your first name?”

USER: “John”

BOT: “Sorry to hear that. Bye. Next, please!”

This is what I am going to tell you in this story, where our main character is not the employee but Google Dialogflow.

BOT: Where are you from?

USER: New York City

BOT: Sorry to hear this, bye!

Wait, what?

Imagine that a bot is asking a simple question “where are you from?”. You insert the correct value and BOOOM. The bot suddenly closes the conversation!

Overview

TIP: familiar with Dialogflow? Start from Conversation exits!

We use Google Dialogflow to provide a rich conversation experience to our customers, driving them through different Intents according to some data that we need them to submit and/or modify.

google.com – Intents

To give context to someone not used to Dialogflow, an Intent categorizes an end-user’s intention for one conversation turn. For each agent, you define many intents, where your combined Intents can handle a complete conversation. When an end-user writes or says something, referred to as an end-user expression, Dialogflow matches the end-user expression to the best intent in your agent. 

When an intent is matched at runtime, Dialogflow provides the extracted values from the end-user expression as parameters. 

Each parameter is described by different attributes, forming a parameter configuration. For this article, let’s say that we are interested at some point in the conversation in a parameter called city that has an Entity type @sys.geo-city.

That’s it. We ask for a parameter called city and we expect our user to write a city in his response:

“Los Angeles”

“I live in Los Angeles”

“I was in Los Angeles”

…

are all valid sentences for us: Los Angeles is extracted from the sentence and assigned to the parameter city.

Let’s finish this paragraph with the definition of Context: Dialogflow contexts are similar to natural language context. If a person says to you “they are orange”, you need context in order to understand what the person is referring to.

Conversation exits

Dialogflow has different features automatically provided that help you to create a rich conversation without having to reinvent the wheel.

System entities are one of these features.

Another important one is something called Conversation exits.

Some magic words will stop the conversation:

• “exit”
• “cancel”
• “stop”
• …

It’s impossible to avoid that this behavior happens, you can just apply some custom logic and reply one last time with a custom response.

Intent, parameters, conversation exits: a recap.

So, we are in a specific Intent, asking the user to give us a city name for our city parameter. At the same time, we know that some magic words will stop the conversation.

The Italian city that stops the conversation

Now we have all the elements to introduce the problem described in the article title. Our clients are from Italy, so the chat is in Italian, and 99.99% of the time the user is going to indicate an Italian city to our question.

One day, we had a support request from one of our clients: “When I insert the city where I live, the system replies with “OK, canceled. Goodbye.”.

-.-‘.

Open the logs.

Open the Dialogflow console.

Check the history.

Oh, wait, here it is! The city is called Fermo, and “Fermo!” means “Stop!”.

BOT: Can you please tell me where XYZ happened?

USER: Fermo

BOT: Sure, goodbye.

The user would be something like -.-‘ .

What the Dialogflow Support team told us about it

We tried to reach out to the Dialogflow support team.

This is the most important part of the email:

Conversational exits for Actions on Google(AoG) are implemented on the AoG app’s side and cannot be overridden by Dialogflow.

Unfortunately, our team is unable to assist as your question is more closely related to Actions on Google

What the Actions on Google Support Team told us about it

We can understand that the word “Fermo” is in reference to a city. However, it is also a system level hotword. Unfortunately there is no solution to bypass the system limitation. Please consider maybe using Fermo city / town or something along those lines, however that may not be guaranteed since it is a system hotword. 

Read it again:

there is no solution to bypass the system limitation

Whaaaaat?? What we did

We provide the Dialogflow experience through the PHP Client library.

We created a list, that right now has only Fermo in it, of potentially dangerous cities.

When the user provides a single word response and we are in an Intent where we are asking for a city to extract a value for our @sys.geo-city parameter, we check if this word is in the list.

If so, we wrap the city in a context: “I live in <word>” before sending the response to Dialogflow.

There is no guarantee that this works, according to AoG support team response, but currently, it seems to work just fine.

The post The Italian city that STOPs Google Dialogflow appeared first on L.S..

The Vision API now supports online (synchronous) small batch annotation (PDF/TIFF/GIF) for all features. To do so, the relevant documentation is Small batch file annotation online.

Let’s see how can we do this with PHP.

Context

Having PHP >= 7.4, the packages to require are:

google/cloud-vision
google/cloud-storage

Code

How to upload the file in the storage

Soon.

Text detection

Even with PDFs we are going to use ImageAnnotatorClient, the service that performs Google Cloud Vision API detection tasks over client images and returns detected entities from the images.

Features

In your request you can set the type of annotation you want to perform on the file. You can check the reference or the features list documentation.

Some examples are:

• Face detection
• Landmark detection
• Logo detection
• Label detection
• Text and document text detection
• ..

The post Google Vision: detect text in PDFs synchronously with PHP appeared first on L.S..

What you should know if you are reading this

Our goal is to have a look at the mobile app development technology for startups in 2021, in order to publish an app for Android and an app for iOS with the perfect trade-off between quality and time+resources. How do we choose the best technology to accomplish this task?

First of all, the most accurate answer is:

it depends.

Every startup has different needs, in terms of business, technology, resources (how many developers? and how many of them are senior?) and deadlines.

Assumptions

There are 200000 articles out there about this subject.

This is why I want to list some boundaries that will “limit” our research. So this is not a general article Native vs Cross platform but is a study for a specific case.

That said, let’s start with these assumptions:

• The start-up technological stack has a very strong connection with Google, mostly with Firebase. This is a very important aspect. In the start-up X the most important technology could be Y, so adjust the content of the article accordingly;
• The start-up was providing all the services in web pages only, and the current app is basically a Webview showing an ad-hoc version of the website. I know, I know! Anyway, there is an ongoing process to transform all the services into a full set of (modularised) REST APIs that we can then consume in our app;
• The app does not have heavy processes in terms of device resources, does not do any image/video processing and so on. Anyway, it has to have reliable access to the camera and retrieve the accurate users position through GPS;
• The current technology is React Native, so we already saw both the pros and cons of a not-native approach. But not-native includes a world of solutions, that we will try to consider in this article;
• The strong connection between React and Firebase, to build business web applications in JavaScript with all the power of the Firebase services, is very well explained in a book that I strongly suggest;
• last but not least, I am not vertical on mobile technologies, so your feedbacks are very welcome.

Technologies in this study

I like a lot the category names reported in a very interesting article by Ionic: Comparing Cross-Platform Frameworks.

I will use the same labels that are

Hybrid-Native

Shared codebase, plus native code. Basically:

it allows you to program your user interfaces (UI) in one language that then orchestrates native UI controls at runtime.

Ionic article Comparing Cross-Platform Frameworks

Hybrid-Web

One codebase, running everywhere. Your code does not get “translated” in native components, but

the UI components you use in your app are actually running across all platforms

Ionic article Comparing Cross-Platform Frameworks

So basically you can code with “web” technologies (HTML/CSS/JS) and your UI components will be shown on the device.

Technologies and platforms in the article

According to the previous definitions, we will analyze the following platforms/technologies:

• 100% native code: one app for iOS, one app for Android, with their respective languages, IDEs and so on;
• ReactNative (Hybrid-native);
• Flutter (Hybrid-native);
• NativeScript (Hybrid-native);
• Xamarin (Hybrid-native);
• Ionic (Hybrid-web) ( – that includes PhoneGap / Cordova).

Image from aalpha

What’s the study about and how do we try to understand if a choice is better than another one

We will briefly describe pros and cons of every technology.
Then we will compare them at high level and a bit in terms of performances.

At this point we will take into account other important factors:

• Who’s behind them?
• The community around each technology;
• Available libs: active development, time to “port” new features, stability, support and community;
• Trends: we don’t want to adopt a technology that is going to become marginal in the next few years;
• Very few observations related to the UI/UX aspect.

One by one

100% Native

Basically, we have 2 separated projects here: one for Android, one for iOS.

Thanks to the respective IDE (Android Studio and XCode) you will probably be able to build a good skeleton of your app and a good enough flow between the views with no or few code required, with a “drag & drop” approach for elements (buttons, text fields, …). A very good preview is also available, without using a simulator, for example.

XCode interface builder from apple.com

That’s great. Anyway, all the logic such as performing API calls, dynamically adding/removing content, handling Push notifications, accessing GPS position, accessing camera/gallery and so on, are performed with a platform-specific language: Swift (Objective-C) for iOS, Kotlin (Java) for Android.

All of this just to say that the two apps will live on two separated, parallel projects. You can of course merge some high-level logic regarding how to manage certain flows, but at the end you have to code them separately.

Two separated projects, two separated teams with separated skills. In the worst-case scenario this could double the needed resources.

On the other side you have the maximum reliability when you are going to handle device resources such as the camera or the GPS.

You have all the debugging and profiling power you need in your IDE.

Even regarding 3rd-party libraries (of course here a lot depends on the library itself) you can have less surprises if you are using the specific library for the given platform.

– Firebase

In case of Firebase, an Android native solution is incredibly good, of course, due to the fact that we’re talking about Google solutions.

Also Firebase has an official SDK for iOS, so the native solution is incredibly good for iOS, too.

Hybrid-web

We have just one project here: you will code an HTML/JS/CSS solution, that’s it. All the UI is shared between the platforms, so you have a very consistent UI and you don’t need to have different components for different platforms. You can, or you can simply have a different CSS applied for each platform.

For debugging purposes you have different layers to take care of, but most of the job related to the UI can be done in a web-based debugging tool like the Chrome Developer tools. One place, different platforms.

So, we were saying that we have just one project here, and this is something to carefully think about if you are a start-up. Additionally, you can “re-use” web related skills in your team to build your app, substantially decreasing the resources needed to complete it.

How about the device resources?

For the most common tasks such as taking a picture or accessing the GPS location, you can be sure to find a plugin (official or not) that takes care of these tasks for you.

The main idea is that there is a sort of Javascript API system to communicate with the underlying platform.

Ionic

With Ionic you can be sure to have someone in your team that can build a simple app with a small learning curve, due to the fact that it supports the major Javascript frameworks and libraries such as React, Angular and Vue.

It has 120 native device plugins for Camera, GPS, Bluetooth and so on.

It relies on Capacitor or Cordova to execute your web components within wrappers targeted to each platform, with API bindings to access device’s capabilities.

This is the architecture compared to the native one:

Native vs Ionic apps. From the official Ionic doc.

– Firebase

The Ionic official documentation points to a specific plugin page on GitHub.

This plugin page has a build: failing tag on it (3rd of January 2021): no good.

Let’s see the popularity of the plugin:

Anyway, if your choice is Ionic and you want to continue your journey with React, I strongly suggest this book to give you very useful hints on how to build robust web applications with React and Firebase.

For this part, that’s it! In the next part we will focus on the Hybrid-Native solutions.

The post Choosing the technology behind a mobile app for a startup in 2021: Native vs cross-platform (part 1) appeared first on L.S..

2020-12-11 18:39:36.965 14746-14746/your.bundle E/RNFirebaseMsgReceiver: Background messages only work if the message priority is set to 'high'

This is something you can find in your adb logcat output when sending a push notification (cloud message) and your app is in background.

More specifically, the problem is mostly related to the idle status of your device.

This is explained in a dedicated Firebase doc, particularly in the “Using FCM to interact with your app while the device is idle” section.

If you are sending a message with curl, Postman and so on, you can add something to your JSON in order to set priority.

So a JSON body like this:

{ 
  "to": "token_here", 
  "notification": { 
    "title": "hey from Lorenzo", 
    "body": "great article dude!" 
  }, 
  "data": { 
    "body": "normal priority notification", 
    "title": "here we are", 
  }
}

becomes:

{ 
  "to": "token_here", 
  "android": {
        "priority": "high"
  },
  "priority": 10,
  "notification": { 
    "title": "hey from Lorenzo", 
    "body": "great article dude!" 
  }, 
  "data": { 
    "body": "normal priority notification", 
    "title": "here we are", 
  }
}

When performing your POST requesto to https://fcm.googleapis.com/fcm/send please remember to add a Content-type: application/json header and a Authorization header with value key=<server_key> that you can retrieve from the “Cloud Messaging” section of your project settings in the Firebase console.

The post Firebase, cloud messaging and problems receiving background notifications appeared first on L.S..

Even if currently it’s a beta feature, you can create multiple versions of your agent and publish them to separate environments.

The problem is that the REST URI bindings do not appear to have been updated in the API definition for this feature.

I wrote an issue in the cloud-php Github project, and John Pedrie, working for the project, asked me to try the gRPC way.

Never did it before, so here’s a guide to help you just in case you are in my situation…hopefully it will be integrated soon!

Assuming you are working with PHP7.3 (and a Debian-based Linux server):

Based on the Install gRPC for PHP doc, on our server (running Ubuntu server 19.10, PHP7.3) I had to install some packages:

sudo apt-get install autoconf libz-dev php-dev php-pear
sudo pecl install grpc
sudo pecl install protobuf

Let’s modify the /etc/php/7.3/fpm/php.ini file adding the following lines:

extension=grpc.so
extension=protobuf.so

Restarting php7.3-fpm:

sudo systemctl restart php7.3-fpm.service

And adding the "grpc/grpc": "^v1.1.0" line to the project composer.json.

Now it’s time to change the code. The usual way is something like:

$sessionClient = new SessionsClient();
$session = $sessionClient->sessionName($projectId, $sessionId);
$response = $sessionsClient->detectIntent($session, $queryInput);

And it has to be changed with something like:

$session = $this->_getSessionName($sessionId);
$sessionsClient = new SessionsClient();
$response = $sessionsClient->detectIntent($session, $queryInput);

// WHERE _getSessionName() is:
private function _getSessionName($sessionId)
{
    $projectId = $this->_getProjectId();
    $environment = $this->_getEnvironment();
    return "projects/{$projectId}/agent/environments/{$environment}/users/-/sessions/{$sessionId}";
}

And that’s it.

In the Dialogflow History tab there is still no way to filter conversations by Environment, but there is an Environment indicator in the conversation Detail.

The post Dialogflow API, support for versions and environments appeared first on L.S..

You can get 2 free months of Skillshare Premium, thanks to this referral link.

With Skillshare Premium you will:

• Get customized class recommendations based on your interests.
• Build a portfolio of projects that showcases your skills.
• Watch bite-sized classes on your own schedule, anytime, anywhere on desktop or mobile app.
• Ask questions, exchange feedback, and learn alongside other students.

That’s it!

• Get inspired.
• Learn new skills.
• Make discoveries.
• Be curious.

I will be more than happy if you can join my classes and give me some feedbacks. The classes are about almost everything that I do in my life (you can check again my skills on the homepage)!..

The post 2 free months of Skillshare Premium! appeared first on L.S..

You woke up one day and your Android phone contacts disappeared.

You try to perform a Google Search but you only find strange solutions, mostly related to some Samsung stuff, factory resets, uninstalling some app, etc…

Suddenly you realise that your Google Contacts was empty!

Ok, too much context, the solution.

Open your browser to https://contacts.google.com/.

In the top right corner you have a gear, click on it. Then click on Undo changes:

Now you can chose the version of your contacts that you want to restore: minute ago, hours ago, days ago, months ago, years ago….

And that’s it, the time to sync your phone and they’re back!

The post Restore your Google Contacts (for your Android phone, too) appeared first on L.S..

Visualizza questo post su Instagram

. God bless Birmingham. In the 20 minutes we filmed Ryan on this bench passers-by gave him a hot drink, two chocolate bars and a lighter - without him ever asking for anything.

Un post condiviso da Banksy (@banksy) in data: 9 Dic 2019 alle ore 5:15 PST

Introduction: Node + express + body-parser + Shopify?

Are you using Node/express/body-parser and Shopify, and (you would like to use) its webhooks?

If the answer is YES to the questions above, I am going to suggest here a way to solve a common problem.

NOTE: This and much more is covered in the Coding Shopify webhooks and API usage: a step-by-step guide class on Skillshare!

The problem

Body-parser does an amazing work for us: it parses incoming request bodies, and we can then easily use the req.body property according to the result of this operation.

But, there are some cases in which we need to access the raw body in order to perform some specific operations.

Verifying Shopify webhooks is one of these cases.

Shopify: verifying webhooks

Shopify states that:

Webhooks created through the API by a Shopify App are verified by calculating a digital signature. Each webhook request includes a base64-encoded X-Shopify-Hmac-SHA256 header, which is generated using the app’s shared secret along with the data sent in the request.

We will not go into all the details of the process: for example, Shopify uses two different secrets to generate the digital signature: one for the webhooks created in the Admin –> Notifications interface, and one for the ones created through APIs. Anyway, only the key is different but the process is the same: so, I will assume that we have created our webhooks all in the same way, using JSON as object format.

A function to verify a webhook

We will now code a simple function that we will use to verify the webhook.

The first function’s param is the HMAC from the request headers, the second one is the raw body of the request.

function verify_webhook(hmac, rawBody) {
    // Retrieving the key
    const key = process.env.SHOPIFY_WEBHOOK_VERIFICATION_KEY;
    /* Compare the computed HMAC digest based on the shared secret 
     * and the request contents
    */
    const hash = crypto
          .createHmac('sha256', key)
          .update(rawBody, 'utf8', 'hex')
          .digest('base64');
    return(hmac === hash);
}

We use the HMAC retrieved from the request headers (X-Shopify-Hmac-Sha256), we retrieve the key stored in the .env file and loaded with the dotenv module, we compute the HMAC digest according to the algorithm specified in Shopify documentation, and we compare them.

We use the crypto module in order use some specific functions that we need to compute our HMAC digest.

Crypto is a module that:

provides cryptographic functionality that includes a set of wrappers for OpenSSL’s hash, HMAC, cipher, decipher, sign, and verify functions.

NOTE: it’s now a built-in Node module.

Retrieving the raw body

You have probably a line like this one, with or without customization, in your code:

// Express app
const app = express();
app.use(bodyParser.json());

So your req.body property contains a parsed object representing the request body.

We need to find a way to “look inside the middleware” and to add somehow the information regarding the request status, using our just-defined function: is it verified or not?

Now, we know that the json() function of body-parser accepts an optional options object, and we are very interesting to one of the possible options: verify.

The verify option, if supplied, is called as verify(req, res, buf, encoding), where buf is a Buffer of the raw request body and encoding is the encoding of the request. The parsing can be aborted by throwing an error.

That’s it, we are going to use it like this.

Let’s change the json() call in this way:

// Express app
const app = express();
app.use(bodyParser.json({verify: verify_webhook_request}));

And we create the function verify_webhook_request() according to the signature documented before:

function verify_webhook_request(req, res, buf, encoding) {
  if (buf && buf.length) {
    const rawBody = buf.toString(encoding || 'utf8');
    const hmac = req.get('X-Shopify-Hmac-Sha256');
    req.custom_shopify_verified = verify_webhook(hmac, rawBody);
  } else {
    req.custom_shopify_verified = false;
  }
}

Basically, we check if the buffer is empty: in such case, we consider the message not verified.

Otherwise, we retrieve the raw body using the toString() method of the buf object using the passed encoding (default to UTF-8).

We retrieve the X-Shopify-Hmac-Sha256.

After we have computed the verify_webhook() function, we store its value in a custom property of the req object.

Now, in our webhook code we can check req.custom_shopify_verified in order to be sure that the request is verified. If this is the case, we can go on with our code using the req.body object as usual!

Another idea could be to stop the parsing process: we could do this throwing an error during the verify_webhook_request function.

Conclusion

Please leave your comment if you want to share other ways to accomplish the same task, or generically your opinion.

NOTE: Remember that this and much more is covered in the Coding Shopify webhooks and API usage: a step-by-step guide class on Skillshare!

The post Node, body-parser and Shopify webhooks verification appeared first on L.S..

Questa guida è stata testata su:
Bionic Beaver

Tempo fa avevo scritto questo articolo, in cui indicavo i passi per l’installazione dello scanner in ogetto .

Recentemente ho fatto l’aggiornamento da Xenial Xerus a Bionic Beaver, e nel seguire le indicazioni presenti nell’articolo indicato prima, lo scanner ancora non veniva rilevato, cosi facendo la ricerca nel wiki italiano (qui la guida), ho scoperto che vanno aggiunti dei passaggi aggiuntivi, quindi riepilogando:

• scaricare i driver a questo indirizzo;
• installare i tre pacchetti, iscan-data_[versione corrente]_all.deb, iscan_[versione corrente]~usb0.1.ltdl7_i386.deb e iscan-plugin-gt-s600_[versione corrente]_i386.deb, oppure lanciare lo script install.sh presente all’interno della cartella (prima va reso eseguibile)
• installare i pacchetti libltdl3, libsane-extras e sane;
• editare il file /etc/sane.d/dll.conf e commentare la riga #epson;
• editare il file /etc/udev/rules.d/45-libsane.rules e aggiungere le seguenti righe:

# Epson Perfection V10
SYSFS{idVendor}=="04b8", SYSFS{idProduct}=="012d", MODE="664", GROUP="scanner"

• editare il file /etc/udev/rules.d/79-udev-epson.rules e aggiungere le seguenti righe:

# chmod device EPSON group
ATTRS{manufacturer}=="EPSON", DRIVERS=="usb", SUBSYSTEMS=="usb", ATTRS{idVendor}=="04b8", ATTRS{idProduct}=="*", MODE="0777"

• digitare il seguente comando, in base all’architettura del proprio pc:

Per architettura 32 bit sudo ln -sfr /usr/lib/sane/libsane-epkowa* /usr/lib/i386-linux-gnu/sane
Per architettura 64 bit sudo ln -sfr /usr/lib/sane/libsane-epkowa* /usr/lib/x86_64-linux-gnu/sane

• infine digitare il seguente comando per riavviare udev:

sudo udevadm trigger

Introduction: Firebase CLI

According to the github page of the firebase-tools project, the firebase cli can be used to:

• Deploy code and assets to your Firebase projects
• Run a local web server for your Firebase Hosting site
• Interact with data in your Firebase database
• Import/Export users into/from Firebase Auth

The web console is fine for a lot of stuff.

But let’s see what we can do with the CLI, and let’s check what we CANNOT do without it.

Installation

Ok, easy.

Assuming you have npm installed, run the following command:

npm install -g firebase-tools

You will now have the command firebase, globally.

Now, run:

firebase login

to authenticate to your Firebase account (this will open your browser).

Firebase tasks

Seleting projects

List your firebase projects:

firebase list

You can then select one of them with

firebase use <project>

Cloud Functions Log

A common thing you would like to do is looking at the logs.

This is possible with:

firebase functions:log

At this point I think there is no way to “listen” for changes in the log, that would be a very useful feature.

The

gcloud app logs tail

sadly does not help here, even if the current selected project is the firebase one.

If you have some tips about it, I will be more than happy to edit this article.

Configuration

The functions configuration is handled through some commands:

firebase functions:get|set|unset|clone

to retrieve|store|remove|clone project configuration (respectively).

Emulate locally

Running

firebase functions:shell

will let you choose which of your functions to emulate locally.

You can then choose the method (get/head/post/put/patch/del/delete/cookie/jar/default) and the test data to start the emulation.

Delete functions

You can then delete one or more cloud functions with:

firestore functions:delete <function_name>

Deploy and serve locally

To serve your project with cloud functions locally, you can run:

firebase serve

When you’re ready to deploy your firebase project, you can run:

firebase deploy

Accounts management

With the firebase cli you can both import and export Firebase accounts with the:

firebase auth:import|export <file>

Surprisingly enough, import import accounts from a file, and and export will export accounts to a file.

Database

I will skip this part. It’s very well documented everywhere in the firebase ecosystem.

Firestore

Here we are.

What we can do with Firestore with our CLI? Almost nothing, I am afraid.

Right now you have only two things we can accomplish.

But we can use gcloud and gsutil to perform other operations, lick exporting / importing data.

Be sure to be logged in with

gcloud auth login

If you are already logged in with different accounts, you can list the accounts with

gcloud auth list

and select one (if the one you need is not the active one) with

gcloud config set account myaccount@gmail.com

At this point, let’s select our project.

gcloud projects list

to list all the projects and

gcloud config set project project_name

to select our project.

Exporting data

First of all, we need a Storage Bucket.

gsutil ls

to see the list of possible buckets to use.

Let’s say that we want to export our data to gs://myproject-abcd.appspot.com/ :

gcloud firestore export gs://myproject-abcd.appspot.com/

You can even export only some collections:

gcloud firestore export gs://myproject-abcd.appspot.com/ --collection-ids=[COLLECTION_ID_1],[COLLECTION_ID_2]

Importing data

You can imagine it, right?

gcloud firestore import gs://myproject-abcd.appspot.com/2019-12-23T23:54:39_76544/

The operation takes some time (proportional to the data size that has to be imported). You can safely close your terminal, the operation will continue.

Checking operations status

Import and export can take time (import takes time even with a very small database).

To list the operations:

gcloud firestore operations list

And you will see something like:

done: true
metadata:
  '@type': type.googleapis.com/google.firestore.admin.v1.ImportDocumentsMetadata
  endTime: '2019-12-23T15:48:03.747509Z'
  inputUriPrefix: gs://myproject-abcd.appspot.com/2019-12-23T23:54:39_76544
  operationState: SUCCESSFUL
  progressBytes:
    completedWork: '2601'
    estimatedWork: '2601'
  progressDocuments:
    completedWork: '8'
    estimatedWork: '8'
  startTime: '2019-12-23T15:47:25.089261Z'
name: projects/myproject-abcd/databases/(default)/operations/AiAydsadsadsadsadVhZmVkBxJsYXJ0bmVjc3Utc2Jvai1uaW1kYRQKLRI
response:
  '@type': type.googleapis.com/google.protobuf.Empty

Indexes (read)

You can look at your indexes:

firebase firestore:indexes

This will list the indexes in a JSON-form of array of objects described as the following one:

    {
    "indexes": [
      {
      "collectionId": "<collection_id>",
      "fields": [
        {
          "fieldPath": "<field_name_one>",
          "mode": "ASCENDING|DESCENDING"
        },
        {
          "fieldPath": "<field_name_two>",
          "mode": "ASCENDING|DESCENDING"
        }
      ]},
      {..}
    ]}

We cannot perform other operations over indexes.

Collections and documents: delete recursively

It’s easy from the web console to delete a document, and it’s easy to do it programmatically.

It’s easy on the CLI, too.

But the nightmare of the web console (and of the programmatic approach, too) is that it does not exist a simple and fast way to recursively delete a collection.

This is luckily possible with the CLI.

You can recursively delete a collection by using:

firebase firestore:delete --recursive <collection>

Bonus: delete all collections

Now, assuming you are testing something and your firestore is full of garbage, you might want to start from scratch deleting every collection.

This is possible running:

firebase firestore:delete --all-collections

If you are looking at your database in the firebase console, please remember to refresh if the UI is not updated (that means that you still see the root collections).

Conclusion

This concludes the article for now.

I hope that the Firestore-side will be developed with other features and commands, because right now is very limited.

One of the most feature I can think of, generally speaking, would be the chance to “tail” logs in the shell.

I would be more than happy if someone can integrate with useful tools and additional stuff.

The post Exploring Firebase CLI with some Firestore tips appeared first on L.S..

The problem: setting max_execution_time / using set_time_limit() I don’t get the wanted results

If you are here, you are probably looking for an answer to the question:

why set_time_limit() is not working properly and the script is killed before?

or

why ini_set(‘max_execution_time’, X), is not working properly and the script is killed before?

The answer is not exactly one, because there are many variables to consider.

The concepts: PHP itself vs FPM

Anyway, I will list basic concepts that will help to find the best path to your answer.

The first, brutal, note is: directives you specify in php-fpm.conf are mostly not changeable from ini_set().

Please also note that set_time_limit() is a almost nothing more than a convenience wrapper on ini_set() in the background, so the same concept applies.

For simplicity, in this post I will talk about set_time_limit(), but the same applies to ini_set(whatever_related_to_time).

According to what we already said, it’s important to understand that set_time_limit() can only “overwrite” the max_execution_time directive, because

they only affect the execution time of the script itself

(there is a specific note in the official documentation).

On the other hand, directives like request_terminate_timeout are related to FPM, so we are talking about the process management level here:

should be used when the ‘max_execution_time’ ini option does not stop script execution for some reason.

Web server dependent directives

What we have seen so far aren’t the only variables in the game.

You should check your web server configuration, too, to be sure nothing is playing a role in your timeout problem.

A very common example is the fastcgi_read_timeout, defined as:

a timeout for reading a response from the FastCGI server. The timeout is set only between two successive read operations, not for the transmission of the whole response. If the FastCGI server does not transmit anything within this time, the connection is closed.

So…what to do?

The basic answer is: have perfectly clear what was defined by web server (ex.: fastcgi_read_timeout), what defined by FPM (ex.: request_terminate_timeout) and what was defined by PHP itself (ex.: max_execution_time).

For example, if request_terminate_timeout is set to 1 second, but max_execution_time is set to 2 seconds, the script will end after 1 second (so, whichever comes first, even if you have set max_execution_time with ini_set() in your script) .

A possible solution is simply using max_execution_time in place of request_terminate_timeout (but use it, too, to define a solid maximum upper-bound).

The max_execution_time description in the official documentation has an hint about this common problem, that is:

Your web server can have other timeout configurations that may also interrupt PHP execution. Apache has a Timeout directive and IIS has a CGI timeout function. Both default to 300 seconds. See your web server documentation for specific details.

Any hints to share?

The post Handling execution time for PHP (FPM) appeared first on L.S..

Listo 👍👍👍#UBports #Ubuntu #UbuntuTouch pic.twitter.com/QANlGJcGYD

— Rosa Guillén (@novatillasku) August 28, 2018

Still Alive!

"Il bello della sconfitta sta innanzitutto nel saperla accettare. Non sempre è la conseguenza di un demerito. A volte sono stati più bravi gli altri. Più sei disposto a riconoscerlo, quando è vero, quando non stai cercando di costruirti un alibi, più aumentano le possibilità di superarla. Anche di ribaltarla. La sconfitta va vissuta come una pedana di lancio: è così nella vita di tutti i giorni, così deve essere nello sport. Sbaglia chi la interpreta come uno stop nella corsa verso il traguardo: bisogna sforzarsi di trasformarla in un riaccumulo di energie, prima psichiche, nervose, e poi fisiche." (Enzo Bearzot)

 

UbuCon Europe

Il mio intervento

Resoconto e live tweeting