Radio Frequency and Grapes!

This is one of my favorite stories about my career as a software engineer, and it happened during my very first professional experience. I never expected it to be the best one, or one of the happiest career moments of my life in terms of professional fulfillment. I believe one factor that favors this is that beginnings are always enchanting — they offer exploration and discovery — and another factor is finding people with synergy and similar purposes. With that, you can build a good culture and real friendships, genuine human interactions that bring joy.

But I bet you clicked here wanting to know what the connection between Radio Frequency and grapes is, right? It's a long story, but an interesting one.

Once upon a time...

It was 2012, and I was in college, in the third semester of my Information Systems degree. I was thoroughly hooked on this new universe of software engineering I'd discovered, wore a beat-up pair of All Stars in that 80s/90s hard rock style, and figured I'd become a millionaire by building a social network like Zuckerberg had done with Facebook. (Spoiler: I did not.)

At that point I already knew basic programming logic and algorithms using Pascal, and had just learned to program in C using the old Code::Blocks. I had an eternal loan from the college library (paying late fees every week) of the book "C Completo e Total" by Herbert Schildt — I carried that book in my backpack everywhere, and it was my foundation for entering this world.

The Internship

Back then, landing a software engineering internship was no easy task — job/internship sites weren't as developed as they are today, and most companies didn't post openings through those channels. The most common thing for me was to go, every week, in person, to a building near campus in downtown Recife. It was an institute meant to connect companies and students, and I remember we'd ride up in an old elevator with a wooden door and another half-iron gated door. Back then there was still a person who sat inside the elevator on a little stool, whose job was to give directions, press the buttons, and take you to the right floor — like an elevator pilot. The elevator's automation slammed the door shut with such force it was startling (BAHHHHHHHHHHHH). Then the elevator would head to the 7th floor, shaking like it was in severe airplane turbulence. Arriving there, the door would blast open once more (BAHHHHH) and you'd step out, thrilled to have survived.

Inside the room there was a wall with physical papers pinned up, listing internship openings, like this example photo below:

If I remember correctly, each paper had a number or some identifier. If you were interested and met the requirements for the position, you'd go to the next room carrying the paper's identifier.

"Fortunately," I never landed an internship that way. It was also a common practice to search Google for companies in the city, and that's how I found a company called IFICOMM Tecnologia, and with not much hope, I sent my résumé to the email listed on their contact form. To my surprise, a few days later I got a call about an internship opening — but at a different company with a similar name: NIXCOMM.

On the phone was Agenor Mota, better known as Mota. He would become my first mentor in the tech world. He owned Ificomm, but was recruiting for Nixcomm, with whom he had a partnership. Nixcomm was, up to that point, a company focused on telecommunications, servers, VOIP, and structured cabling — no software at all! The company's founder is one of the craziest and smartest guys I've ever met — an electrical engineer who loved Pink Floyd and had a vision for innovation and business that was way ahead of its time. That was the beginning of my story as a software engineer.

Hermes Aguiar, the Pink Floyd fan, wanted to create a new department within the company. He was putting together a small team of 3 people, with Mota in charge, to build software products based on RFID (Radio Frequency Identification) technology. I had never heard of it before, and that made everything fascinating. Pure adrenaline!

Funny thing — I've always been a night owl, spending my nights studying and sleeping a lot during the day. When Mota called, it was noon, and I had just woken up. Groggy-voiced, I answered the phone yawning... and in the end, I still got picked! Thanks, Mota!

Nixcomm, my second home, where it all began.

A week later, there I was at the office, about to meet the third person in this story, my coding partner: André Andrade (the second intern picked). The company had set up a room for software development alongside the project team (the folks who used AutoCAD to draw floor plans), and over in the corner sat all the hardware, brand new!

Antennas, controllers, handheld readers, desktop readers, coaxial cables, spools of wire, kiosks, computers, monitors, a graphics card, driver installation CDs (yes! CDs!) — everything brand new! It was like a kid on Christmas morning, unboxing everything and plugging in every cable, every connector — my god, it was magical! The year was 2012... it was still normal to have square (4:3) monitors, and Intel had just launched Ivy Bridge, 3rd generation. There was a giant server next to my desk, with lots of blinking lights and a ton of cables organized just as structured cabling certifications require. It was a delight for an IT professional starting out at the time.

This is where, without meaning to, I also dove into the world of Hardware, building embedded software in my very first professional experience — a powerful turning point in my life.

Since we were a small team, we had a full end-to-end experience of the business, working through the entire chain — not just writing code, but buried in hardware documentation, calling RFID equipment suppliers, purchasing equipment and RFID tags in their many different types (there are quite a few, each for a specific situation), talking with the project team to figure out where the equipment and cables would go on the floor plans, and talking with the operations team to understand the metrics for how many items we'd need to roll the solution out across different locations with different requirements. I loved this job so much that sometimes I'd think: "Am I really getting paid for this?"

As time went on, we gained more experience with programming and RFID, and we built several solutions with RFID technology (which I'll cover in other posts). I also played designer during this time — I drew the layouts of our software products, the brands and visual identities for the products (several of which we patented), and even the commemorative t-shirts. These projects felt like children to us, going from the immaterial realm of our sketches on the central table and technical discussions, all the way to the code that brought the equipment and environment to life. After a year as an intern, we got hired and were given a brand-new room dedicated to Nixcomm's Software project.

We had our own version of Kanban, on a physical board on the wall of the room, where we built everything in whatever way worked best for the team and delivered results in the end.

We did a lot of prototyping, sticking screens and ideas on the wall, organizing by flow and how ideas fit together, and we'd sit for minutes in front of that wall discussing and checking every detail, chin in hand like philosophers of ancient Greece.

One day Mota brought us a giant poster to put on the wall, with the most commonly used types and namespaces in .NET Framework 3.5. It was awesome!

We had a lot of events during the year, and it was tradition to have a t-shirt for each one. Internal parties, client parties, charity events, the World Cup, and so on! Here's the t-shirt from Brazil's World Cup campaign — I still have it today.

A beautiful friendship and a wonderful culture had developed within that team! Affectionately known as: Equipe Cão (the "Dog Team")!

The funniest part is that both Hermes and Mota had very open minds about work models for that era (2012). Nobody cared what time we got to the office, what time we left, or whether we'd work on some software module from home instead of coming in on a given day. We had the office key and could simply come and go whenever we wanted. Even with that freedom, the culture and the work were so enjoyable that we'd often work many hours past closing time without even noticing time pass — it was like a mad scientist's lab, where we got to play with laser beams and program the monster!

When we got our new room, we set the desks facing a huge window, which was gorgeous at sunset!

Here's a photo of André and me integrating desktop Windows Forms .NET Framework 3.5 modules with ASP and Windows CE for mobile devices, pocket PCs. (Old school.)

It was a successful, high-performing team! No solution was impossible. And that partnership grew into a great friendship that lasts to this day, whenever we remember those moments and crack up about the Equipe Cão!

Ok Marcos! Cool, cool... And the grapes?

Now let's get to the project that gives this post its title. We closed a contract to build a solution for a grape farm. The project aimed to measure the efficiency and productivity of the teams along their production lines. We'd use RFID tags and antennas mounted on the farm's conveyor belts to read every tag on every pack of grapes as it passed through each stage of the process, creating a timestamp for each reading — almost like an SLA (Service Level Agreement) for grapes.

The project was extremely challenging — the technical bar was very high, since it combined a lot of knowledge we'd built up so far, plus things we hadn't done yet. In the world of custom engineering solutions, there's always a surprise; there are no case studies to draw from or code samples to look at. We really had to build everything from scratch, with a lot of creativity to solve complex problems with unusual solutions.

The industrial environment challenge

The difficulty was mainly compounded by the industrial environment — the site where the solution would be deployed was full of conveyor belts, machines, transformers, and various power sources that could interfere with the radio frequency, plus people constantly walking around. The network was local-only and intermittent, so we already anticipated the challenge of syncing data between the equipment at location A and the database server that would sit at location B, far away, but still within the farm.

We would need to build:

• An embedded RFID hardware solution that could run on its own, with nobody operating it.
• Something light enough to run on a small-form-factor business Windows PC with fairly modest hardware.
• A way to manage data offline with syncing to the server whenever there was network connectivity, due to the intermittent connection.
• On top of that, automatic backup structures for the data in case of failures, power outages, or whenever the site went without a network for several days.
• And our project team had to figure out the routing of power and network cables across the warehouse floor plan, together with the field team who'd handle the electrical installation and mount the equipment in the right spots.

No doubt about it, this was a massive engineering challenge — I believe one of the biggest I've ever faced. A problem like this isn't solved with code snippets you find on the internet today. Back then, there was no ChatGPT, nobody was even talking about AI yet, and there were no boilerplate projects to base things on — everything was unique to that situation, that company, that project.

Equipment and Resources

After understanding the solution's goals and expected results, the immaterial phase of the project began: think, think, and think some more.

Questions kept coming up:

• Which equipment would be most efficient for the solution? What kind of antennas, which readers? And what frequency?
• What kind of tags would make the most sense for an industrial environment, able to withstand humidity, dust, human handling, and so on.
• Which database could we use for an on-site solution with constant writes — relational or non-relational?
• How would we architect the software project? What layers? What patterns should we use?

There were a lot of questions, and in my opinion, that's the most important moment in an engineer's career, no matter the field. You need to develop the ability to imagine things in the world of ideas first, and then bring them into matter. That's being a pontiff of sorts — head in the clouds, feet on the ground, connecting both worlds.

For the readers, we decided to go with Motorola models, for their quality and because we already had a lot of experience with those models from previous projects.

FX500 and RD11320

For antennas, we also had 2 models in mind, for different situations at different points in the process.

AN400 and AN480

There were countless RFID tags to choose from. We needed to pick one suited to all the problems we mentioned above, such as: humidity, dust, human handling, frequency, and available memory to record data...

For those who don't know how an RFID tag works, here's a quick explanation:

Most of the solutions we worked on used passive tags, which is the type you see above. This tag has a tiny chip inside that stores the tag's identity information plus a few extra slots we could write data to.

A passive RFID tag has no battery of its own and is powered by the energy of the radio signal emitted by the reader. When it receives that signal, the tag reflects back an encoded response with its identification data.

And the types of tags vary widely in: type, material, signal strength, antenna size, memory capacity...

When we ordered rolls of tags for a solution, we could ask the manufacturer to pre-write IDs within a specific range, or write some specific field we could use as a flag to identify a tag group, project, phase, and so on. Most tags had memory fields protected by a password, so it wasn't enough to just point a reader at the tags and read the data — it was encrypted.

We could also choose the tag's artwork...

Not long after this project, we bought a printer/encoder that let us buy blank tag rolls and write the data and layout ourselves, cutting out a step in the process. It looked roughly like this...

I don't quite remember if this was the exact model, but that's a story for another post...

For the database, we used a mix of SQL Server for the application, replicated to an on-site MySQL for reporting via the client company's internal PowerBI.

As the readers captured tag data, they wrote it to a local SQL Server database on the machine running the software solution, and it synced to a MySQL instance elsewhere on the farm.

For software development we had Motorola's SDK, which was in C# .NET Framework 3.5, with GUIs built in the old Windows Forms and ASP, on the heaviest IDE I've ever seen: Visual Studio (VS Code didn't exist yet), and it was nearly impossible to run C# anywhere other than Microsoft's own IDE. Opening the project, my laptop's fan practically took flight from the workload 😄, even with a last-generation i7.

Everything was more complicated — even just setting up the environment, we had a manual we'd written ourselves, listing which DLLs needed to go in which folders, which version of this, which version of that... Docker was still in its infancy back then and wasn't a reality at local companies yet. So the environment was set up the hard way, following our manual — it was a real soap opera!

And a lot more decisions kept getting made. Sketches, negotiations with suppliers, back-and-forth with the project team, logistics... all of that, the three of us handled — the code was the last part. A few days later, we were prototyping, and we made test tags to simulate the environment inside the office, where the process was: tags moving on a belt, being read and processed. We'd bring those test tags to the pilot project, until the final tags we'd ordered for the official project arrived.

We made the test tags by hand, adding a plastic sleeve to protect the tag from hand sweat and other liquids. Utility knife, printer, paper cutter, tag glue... and there we go! All of this happening inside the office, since the next step was packing our clothes and hardware bags and hitting the road for the pilot project.

Hitting the road

With the prototype ready, we rented a car and hit the road. We went ahead of everyone to get to know the site, and the operations team would arrive later to install the equipment and electrical connections.

It was more than 10 hours of travel, with plenty of Jelly Beans along the way! A real tech road trip.

We arrived at the hotel exhausted that day, but I remember we "arranged" the equipment on a counter the hotel had, hooked things up, and ran a few final tests before starting the installation week. It would be several days, figuring out the best spots to position the antennas, testing the software, connections, integrations, and testing in the real environment.

In no time, the hotel room looked like a war zone, everything scattered around, full of wires, equipment, and tags!

The next day, we woke up early and headed straight to the site — another 50km by car on dirt roads full of pebbles, which made the car feel like it was driving over a hailstorm of marbles, sliding around constantly.

We arrived, industrial environment.

Industrial environment, with all the challenges we'd planned for and arrived prepared for.

Now it was time to unload everything from the car — it looked like we were moving house. We went to the warehouse where we'd run the pilot project. We were seeing the process in person for the first time, watching which conveyor belts the grapes moved through. And at this site, we needed to wear special protective gear.

And little by little, together with the operations team, we started mounting the readers and antennas in the right spots for good reads of the grapes passing through the belts at each stage of the process, each one with its own RFID tag.

The FX500 was mounted in an enclosure, with the antenna cables running out through a side opening so they could be positioned on the belt.

The RD11320, on the other hand, was set up mounted just below the AN400.

Everything mounted and wired up, now it was time to distribute the control tags on the test grapes.

Ok! After a lot of work and analysis, everything was in place. Electrical, check. Now we needed to test the software in the real environment, and of course, "in practice, theory turns out different."

The BUG! 🐞

When we opened the main software that controlled the readers together with the antennas, we noticed unexpected behavior — the kind of problem you can only spot in a real environment.

It was official — we had a bug!

In Motorola's SDK for RFID readers, there was a main loop that served as the central structure responsible for continuously monitoring the environment and capturing the RFID tags detected by the antennas. This loop ran intermittently, meaning it stayed constantly running, performing successive reads of tags within the reader's range.

The software first initialized the reader and configured the read parameters, such as frequency, antenna power, and supported protocols.

The main loop ran indefinitely, calling API functions to perform reads over and over. On each iteration:

• The reader sent out radio signals to power the RFID tags within its coverage area.
• The tags responded with their unique identifiers (EPC – Electronic Product Code).
• The software captured these responses and stored the data in a control structure.

The API also provided mechanisms to notify events, such as: new tag detected, read error, and others.

The bug was in this structure, which, on top of everything, made customizing the behavior quite complex. The SDK's internal code was rigidly closed off to extensions, going against the Open/Closed Principle (OCP) of SOLID. So we had to use the structure exactly as it was built, without much flexibility, which made solving the problem even more challenging.

This error happened occasionally, in specific situations — like when a grape sat under the antenna for too long and the reader also picked up a grape on the belt next door, or when two or more grapes passed simultaneously under the reader while two or more others were on the neighboring belt. It was a tricky problem. Even though we knew that, being a pilot project built from scratch for a one-of-a-kind need, some bugs were inevitable, it was still frustrating to watch them happen. Moments like these called for emotional intelligence.

The bug caused the main loop to get stuck, with the code ending up in a state where it wouldn't correctly process new reads.

It also caused:

Event overload: the system generated repeated events due to successive reads without proper control.

Concurrency issues: if multiple threads tried to access the data at the same time, unexpected behavior could occur.

Timeouts or full buffers: if too many tags were detected at once, the reader could get overloaded and fail to process events correctly.

And this is where we hit our turning point!

An experience that had been incredibly rewarding was starting to become incredibly frustrating. That day we collected all the data we could, and once the plant's workday ended, we went straight to the hotel to find a way to fix the problem. I remember that night, we fell asleep on top of our laptops, and woke up to the first rays of sunlight coming through the hotel room balcony. A hot shower, and back on the road, since the workday at the plant starts early. Another day of testing, debugging, code changes, back to the hotel, pull an all-nighter, catch a nap, and do it all again.

The week had come to an end, and it was time to head back, but the mission wasn't accomplished yet, and the frustration in the air was overwhelming. I remember Mota offered to let André and me fly back home while he stayed behind to solve it alone. It was very kind of him, but we wouldn't accept — we'd all arrived together, and we'd all leave together.

It was the first time I'd ever traveled for work, and the feeling of being far from home, far from my girlfriend (who is now my wife), combined with the frustration of watching a project go wrong, was awful. Up to that point, there had never been a project this team couldn't crack, nor a problem we couldn't overcome.

That day, I remember sitting on a concrete step by the warehouse door and saying, "It's not going to work, I don't know how to fix this." It was a brief moment of despair, until André came over and asked me a few questions:

• Who planned this entire project?

And I answered:

-- The three of us

• And who chose the equipment, and built all the code from scratch?

Once again I answered, loud and clear:

-- The three of us

And then the last question came:

• So who else is going to solve this problem, if not us?

Mental clarity! There it was! I got it! It might seem obvious, but it's exactly these obvious moments that bring on the deepest realizations. From that instant on, our mindset completely changed. It was like ancient war drums, reigniting the warriors' spirit. It was the weekend, and we kept working on the solution. At the hotel, each of us would implement something and share our reasoning with the others, until we could reach a synthesis.

The good camaraderie within the team was essential to solving this problem — everyone was focused as a group, there was no competition, only the goal.

And in the following days, after a lot of teamwork, we did it!!! We nailed it, Brazilian style!!! The bug was fixed!!

The fix involved several factors, such as:

• Creating a processing queue — using a FIFO queue structure to store read events and process them sequentially, avoiding event overload and loop lockups.
• Lowering the read frequency and the reader's power range to avoid excessive detection in scenarios where tags from parallel belts were being picked up.
• Building a cache of active reads, storing recently detected EPCs and discarding duplicates within a defined time window.
• Building more sophisticated logs to pinpoint failures and adjust the reader's parameters as needed.

We went back on the next business day after the weekend and updated the software on the reading units — mission accomplished!

Lessons Learned

What I gained from this experience went far beyond the technical — it was a lesson for life and for my career.

A few things worth mentioning:

• Even if you plan a situation from A to Z, there's always a chance something goes wrong — and that's ok. When we have a controlling personality and believe we're in command of life's situations, we tend to suffer more when things don't go as expected. Adverse experiences are there to bring us unexpected lessons, forging new skills for improvisation and critical thinking to solve problems in moments of crisis.
• Friendship is a powerful foundation for achieving big goals. In a good team, there will always be people with different skills, but when everyone understands their role and uses their strengths to reinforce the group, the team becomes cohesive, without fights over individual spotlight. To draw an analogy with the human body: every cell needs to work together to keep the organism in homeostasis. If a cell starts acting uncontrollably, it can turn into cancer, compromising the whole system.
• In moments of crisis, emotional control will be more effective than raw mental horsepower, because it lets you stay centered, and bring everyone else to that center — the place we need to be in to solve problems during a crisis.
• Last but not least, it's essential to have a devotion to knowledge and always be willing to learn from others. If you don't keep studying, you won't be able to contribute when it matters. And if you're not open to learning, you'll miss the chance to gain knowledge that only someone else can pass on to you.

To become an experienced professional, you need to go through a lot of experiences.

This experience, for sure, was and will always be an internal reference point, whenever I need inspiration or think I can't solve something.

Thank you for the good times, where the work always made sense, André and Mota!