• Original post: https://www.reddit.com/r/pinball/comments/1m9m81p/pinball_2000_development_lore_part_6/

    These are my experiences as part of the Pinball 2000 team. Feel free to ask questions. I’ll gather up multiple answers into one comment like I did with the initial post. Now, without further ado…

    Part 6 – Miscellaneous silly anecdotes

    These are unconnected bits and they’re not in any particular order. I either thought the situation was funny (sometimes only after the fact) or they show what life is like when you’re involved in a project with so much passion and so much riding on the outcome.

    SNACKS!
    It was a lapsed tradition among the pinball programmers to go down to the lunch area in the afternoon, buy things from the vending machines and chit-chat just as a break from the grind. At some point in Pinball 2000 development we revived this, partly because someone left a hand bell up by our offices. Dwight would ring the bell and call out “Snacks! Snacks!” and we’d go. The bell was loud and obnoxious, but people tolerated the noise because most things were manic most of the time. I’d generally have a Diet Pepsi and a chocolate bar. I was averaging six bottles a day so my caffeine levels were high but not quite into heart palpitations territory. One day we were sat there talking and Tom looked over at me. He says “You’re eating around the letters?!” and he was right. I often got one of those Nestle Crunch bars. I would leave the letters until last so I could eat them one at a time and rearrange them into anagrams. Crunch became churn but I forget how it went from there.

    PHONE TAG
    The company phone system wasn’t sophisticated. If you called someone and they were already on a call you’d just get the busy signal. If you needed the person urgently you could call an extension and whatever you said would come out of speakers all over the building like an announcer at an event. You’d hear things like “John McCaffer, please call 123”. People who were more important would get paged like this and you could gauge how busy they were by how many pages they got. Anyway, one evening I was in the bathroom and I heard someone get paged. Then someone else, presumably the person being paged, called in and paged the original person. The first person paged again with “… and please hang up your phone”. The second person paged AGAIN with just “It’s hung!”. I didn’t recognise the names and I don’t think I ever met them.

    RESTAURANT CRITICS
    We had several places we went to eat on a regular basis, for lunch and for dinner. Some of them had nicknames. Sometimes the names registered a person’s dislike of the place, but not always. For example there was a Popeye’s nearby that people called “boneyards”. Apparently one of the videogame teams ate there regularly and were in the habit of throwing chicken bones over their shoulders into the potted plants, which I neither witnessed nor would condone.
    There was a Mexican restaurant called ‘Taco Y Salsa’ that at one time had been a bar and Greg Freres had played a gig there when he was in a band. The staff had trouble with my British accent, but I liked the food. Larry came with us one lunchtime and was unimpressed. At some point after that we were deciding where to go and he called it “Tacos & Horsemeat”. I think we stopped eating there not long afterwards.
    We used to get dinner at this diner called ‘Golden Nugget’. I think there were several in the area, but I’ve no idea if they still exist. I would usually get either a burger or breakfast food. Keith always ordered french toast with a side of sausage. One time Lyman decided to join us, having been skeptical about the place beforehand. I thought their milkshakes were really good so Lyman ordered a chocolate shake. When it arrived the metal cup had a tiny piece of banana on it (I expect from a banana shake being made at the same time). Lyman was very unhappy with this and refused to ever go back to “Dog Nugget” as he subsequently named it.
    Bill got fed up with the usual selection of lunch places and pushed for us to try to somewhere new. It was called ‘Nantucket Fish House’ and it was close by. I know Nantucket is a real place but I have a dirty mind and so I kept thinking about what is usually rhymed with it in limericks. It was small and quiet and nondescript, but unfortunately the food was quite bad. I’m forgiving with most things but if you can’t get fish & chips right when one of the words is in the name of your restaurant, you don’t deserve repeat customers. We never went back and I think Bill went several months before resuming his quest to broaden our palates.

    THE TAPESTRY
    There was a Mexican restaurant down the block from where we worked called La Finca. The food was really good so it was a good place to take new people. There was also an ulterior motive, in the form of an initiation rite of sorts. There was a tapestry on the wall of a step pyramid, like the Aztecs built. One side was partly overgrown and the other was cleared of vegetation. We would ask new people to come up with an explanation why this might be the case, since the job involved creativity. It was fun to hear what people would come up with. Someone said it was because the blood from sacrifices ran down that side. My own answer had been that they were union workers so they were on a mandated break.  Bill was hired before me so I wasn’t present for this, but he told me his answer had been “It’s a beach towel!”

    STEALTH
    I worked very late hours most days. I would arrive around 11:30am, do the things I needed to start the day and then go to lunch with the others. I would usually stay until 2am or even later. It was good to have time when I was on my own because I could avoid being distracted. I only lived five miles away so my evening commute was really easy. One night it was really foggy and there was almost no-one else on the road. I was tired and nearly home when someone on a bicycle came riding through the intersection towards me, on the wrong side of the road. He was a black guy with no lights on his bike, wearing dark clothing. I think the bike was dark grey or black as well. I don’t think he could have been less visible than he was. It was lucky for both of us that I saw him at all.

    REMODELING
    As we got closer to putting the first game into production the factory got involved. They needed one of their engineers to help make sure the games could be built easily. There were often mechanical things that needed to be tweaked. One afternoon I came back from a trip to the vending machine (I drank a LOT of Diet Pepsi) and noticed a new hole in the drywall outside the office of one of the mechanical engineers. The thing is, this hole was at head height. It turned out that the manufacturing engineer had unilaterally decided that the slide rails under the playfield were unnecessary and didn’t want to do anything with them. One of our people had been angry at this and punched a hole in the wall beside this person’s head. The slide rails were added back to the production process.
    This was not the only time tempers flared strongly enough to lead to damage. The building manager tolerated a certain amount of destruction, but would make sure people knew when they were over the line. He’d worked for the company since before I was born and was a good guy. He dealt with a lot of bullshit from over-indulged creative people. He had two workers who would fix damage like this along with making layout changes. One was short and plump and the other was taller and thinner. We nicknamed them the Super Mario Brothers. They would be tasked to reconfigure things all the time. There was an open area with a bunch of chairs and they had to put walls around it. I don’t know who wanted that and none of us pinball people thought it was helpful. I had no problem with them. They were just doing what they were told, but it was sometimes strange to find a door where there hadn’t been one or a corridor that had been extended.

    EXPLETIVES
    I liked my office to be as dark as possible so that the only light game from my PC monitor or the game. I kept the lights off and my door ajar. At one point I had a board with a bunch of flashlamps laid out in two rows, with covers in all different colours. I wrote a little bit of code that would flash them randomly and put them inside my game so that it was like having a mini rave in my office. It must’ve looked odd to people passing my door. The lady who had the office next to mine worked in the drawing office keeping track of blueprints and so on. Her name was Nina and people called her the “purple dragon lady” because she loved purple clothes and didn’t indulge people’s foolishness. I had my little rave running one day and was really fired up working on something. When my code crashed my game I said, loudly, “Goddamn motherfucking piece of shit!”. Nina immediately called out “Coral!” (actually my deadname, but I won’t use that here) in that tone everyone knows as Mom Voice. I popped into her office and apologised, but she was smirking. She told me not to worry because she “had raised two boys”. She was one of my favourite people so I did still watch my language a little more during normal work hours.

    ALL-NIGHTERS
    Some of us would work through the night and into a second day without going home to sleep. This usually only happened when we had a really important thing happening that second day, like a demo for the CEO. One night there were five or six of us and we got hungry, so we drove to the supermarket nearby. Picture a gaggle of slap-happy overtired programmers pushing a cart through the aisles and tossing random junk food into it. There were cookies, marshmallows and so on. I was in the aisle where the refrigerated shelves held baked goods. I noticed a slab of chocolate cake, and signaled Lyman, who happened to be with me. “Are you thinking what I’m thinking?” I said while pointing at the cake. He agreed and we decided to split it 50/50. It was at least 2lbs of cake. I said we should call it “Megacake”. We put it in the cart, I paid for it and when we got back to the office we cut it in half. I ate all of my half of Megacake that night and I think Lyman did the same.

    COMPATIBILITY
    Keith was working on game 4. One day he came by my office to tell me a character in one of the fonts looked wrong. I asked him to draw just that character and scale it up so I could see whether it was the font itself or some problem with the graphics code. He did this and we could both see that one pixel was clearly one position off from where it should be. I’m a decent programmer, but I’m a better debugger. I found the problem, which was in one of the tools that processed art data to go into game ROM. It only happened when the source art, which was saved in a compressed format, happened to trigger a specific part of the decompression code in the tool in certain circumstances. This was why we hadn’t seen the problem before. It was a simple fix, Keith and I verified the problem was solved and I got back to whatever I was working on before. A week or two later Tom came by my office because he’d tried to build a previous version of Star Wars Episode 1 and couldn’t get the checksums from the build to match the ROM image we’d used before. That meant the ROM image was different in some way and that was concerning. Suddenly I remembered that bug fix I’d made and we reverted it locally. Sure enough, now the checksums matched! There was a character in one of that game’s fonts that had the same problem but it was one we’d never actually used (probably a tilde or some other punctuation). I fixed my fix by adding an option to the tool so that you could disable the bug fix and modified the build script for Episode 1 so that it used that option. To this day I’ve never had to write another run-time conditional bug fix.

  • Original post: https://www.reddit.com/r/pinball/comments/1m5bg6c/pinball_2000_development_lore_part_5/

    These are my experiences as part of the Pinball 2000 team. Feel free to ask questions. I’ll gather up multiple answers into one comment like I did with the initial post. Now, without further ado…

    Part 5 – Higher-level support for graphical “display effects” akin to the WPC way of thinking

    There was now a way to use the display as a shared space, but we still needed to convey information and choreograph between multiple users of the display. As well as individual Displayables we needed to group them and hide or show them together.

    The WPC games had a concept of a “display effect” or “deff”. When nothing else needed to use the dot matrix display it would show the score. Something more important, for example locking a ball, could override the score sweep. That deff would play its animation, show a message about the ball being locked and then exit. One deff could pre-empt another. Earning a replay or an extra ball would cancel a less important deff, such as increasing the bonus multiplier. Those were “foreground deffs”. If they were interrupted they would not be restarted. There were also “background deffs” which could be paused by a more important deff and would resume when there was nothing pre-empting them. For example, when you locked a third ball you would get a foreground deff to show multiball was starting. Then a background deff would take over from the default score sweep so it could share details like what to do to score a jackpot. If you shot a jackpot a foreground deff would play and then the multiball background deff would resume. Once you drained back down to one ball, a foreground deff would summarise the multiball. The multiball background deff would finish and the score sweep would resume. The game could have multiple background deffs stacked by priority. For example if a hurryup was active when you started multiball and was still going when you drained out of multiball (maybe the hurryup paused until then), the hurryup background deff would resume until the hurryup ended, and so on. The score sweep was the lowest priority and made sure the display was always being used.

    I knew Pinball 2000 would need a way to do the same things. The jet bumpers might show animations every time the ball hit them and we wouldn’t want stray Displayables to be accidentally left on the screen. We also had to support multiple foreground and background deffs being active at once. The dot matrix display could really only support one rule at a time, but our video display was shared by multiple rules. That made the choreography a bit more complicated, but it was not that different from how WPC games did things. We could keep a prioritised stack of deffs, some of which could be paused and resumed.

    Like there were Displayable items and a DisplayManager drawing them in order each frame, I built the equivalent things for handling effects. There was a DeffManager to control all the active display effects. A rule could ask the DeffManager to start a DisplayEffect and then not have to bother with it any further. When the DeffManager told a DisplayEffect to start, the effect would allocate a bunch of Displayables and register them with the DisplayManager. When the DeffManager told an active DisplayEffect to stop, the effect would unregister those Displayables and free their memory. When a DisplayEffect was pre-empted it could choose to be suspended or just let itself be stopped. When the DeffManager told a suspendable effect to suspend that effect would hide all its Displayables. When the effect was told to resume it would ask the DisplayManager to start showing its Displayables again. The Displayables still had their Z values so an explosion could draw over the top of a building owned by a different effect. If the explosion effect wasn’t important it could be killed off without having to remove the building as well.

    DisplayEffects needed something like a Displayable’s Z value. When a rule asked the DeffManager to start an effect it would give it a priority. Something important like earning an extra ball would have a high priority and something like the jet bumper DisplayEffect would have a low priority. This didn’t fully solve the problem of what to pre-empt though, because something important that didn’t need the entire display wouldn’t want to kill off less important things that didn’t interfere with it. I spent time thinking about this and chose to handle it by adding a threshold value that was given to the DeffManager along with the priority. The threshold meant “anything lower priority than this should be stopped or suspended”. Suspendable effects would mostly have a threshold of zero. They didn’t need to suppress each other for the most part. If multiple rules wanted to display something over a ramp, for example, the less-important effect could have a very low priority and the more-important effect would have a higher priority for itself, and a higher threshold than the other effect’s priority. Whenever an effect started or stopped the DeffManager would go through its list of active effects and find the highest threshold. Any effect with a lower priority than that threshold would be stopped or suspended. Things that needed to commandeer the whole screen, such as multiball start, replay or extra ball would have a high priority and high threshold. Once that sort of effect finished and the threshold was recalculated, any suspended effects could be resumed if they were above the new threshold. All this complicated management was a black box and game programmers just had to set up their priorities and thresholds based on what they felt was most important for the player to see at any moment.

    Around the same time as I was building this system I added another convenience feature for Displayables. I made a new object, a Sprite, which was a Displayable that could be told how to scale or move itself, and/or to show successive image frames over time. The DisplayEffect could create a Sprite, set up its movement and animation and the Sprite would update itself automatically. Every time the DisplayManager started a new frame it would tell every Displayable to update itself. By default Displayables did nothing, but a Sprite would do what it had been set up to do. It could also bounce away or wrap around if it moved to the edge of the display. It could also be restricted to part of the display when behaving that way. It could restart its animation when it reached the end or play it back in reverse, ping-ponging back and forth. This was separate from being drawn so that hidden Displayables would automatically be up to date if they were made visible again later.

    We had very little time to make our first two games and it was very important to me that game programmers would have to spend as little time as possible making their graphics work. The more time they had to work on game rules and cool stuff the better. I was really pleased with the whole display management system.

  • Original post: https://www.reddit.com/r/pinball/comments/1m10n15/pinball_2000_development_lore_part_4/

    These are my experiences as part of the Pinball 2000 team. Feel free to ask questions. I’ll gather up multiple answers into one comment like I did with the initial post. Now, without further ado…

    Part 4 – How we hashed out technical decisions and competing philosophies

    NB: I had a different title for this section originally but I decided it would be more interesting to narrow it down and expand one of the later parts.

    Any collaborative creative project will have a variety of viewpoints and personalities amongst its members and Pinball 2000 was no exception. Tom was the master of deadpan snark. Duncan was calm. I was emotionally reactive and volatile. George inspired passion. JPop appeared whimsical but was actually a very deep thinker. These were the people I worked most with in the early days, but it’s just as true for everyone else who brought Pinball 2000 to life. We all had the same goal, but we had to trust each other a great deal. Sometimes that brought friction. It was especially hard for me because I was not used to working closely with others, and my way of thinking is often quite different from other people. Luckily we all got used to each other’s quirks very quickly.

    Some of the times I remember most clearly from the early days of the project were when Tom and I would be architecting something and being really tough on each other’s ideas. We never got angry at each other, but there were no sacred cows when it came to programming. I loved those times because I could be confident in the quality of my solutions if Tom was happy with them, or I’d learn something (or at least learn what I needed to learn more about) if Tom found flaws in my designs. One example of this is to do with memory management. Any program that asks the OS to allocate memory to it also needs to return that memory to the OS when it doesn’t need it any more. If it keeps allocating memory and not freeing it eventually the memory will run out and the program will crash. That’s called a memory leak. Tom was adamant that we should not let this happen and I agreed with him. Having to reboot a pinball machine is silly. In fact, having to reboot any device that should just stay powered on and doing its thing for weeks at a time is silly, but that’s a rant for another time and place.

    His solution was to track which thread (basically which task in the system) had allocated the memory and if that thread exited or was killed, all the memory still allocated to it would automatically be freed. If a thread itself spawned another thread, that memory was considered allocated by the parent thread. Since almost all threads were killed when the ball drained or when the game ended this was a really good way to avoid memory leaks. My problem was that the display code needed to manage its own memory both for individual Displayables but also for its internal state. I wanted a thread to be able to start up some graphical effect and not have to keep track of it and especially not to have to worry about corrupting memory used elsewhere. We got together in an empty office and talked for a good while. Typical programmer stuff, drawings on whiteboards, back and forth. We came up with a solution that let a thread sever itself from its parent. I don’t remember the details but it was a concise synthesis of several things we had explored

    For WPC games the multithreading was co-operative. Until you exited or slept, nothing else would run. That made most things easy to think about but it had drawbacks. Pinball 2000 had a much more powerful CPU, and its system software was based on a tiny Unix-like OS kernel called PC-Xinu (an acronym – Xinu Is Not Unix – and we called our version Xina – Xina Is Not Apple – referring to the WPC system software. Recursive acronyms are an example of humour in older programmers). That meant pre-emptive multithreading was the default. Tom was extremely knowledgeable about that kind of programming, but I hadn’t done much of it. I had a learning curve to climb as I worked on my parts of the codebase so I didn’t cause deadlocks or corrupt data by accessing it from multiple threads without protection.

    I wasn’t the only one who would have to learn about these dangers. On WPC individual threads were low overhead, their thread IDs were pre-assigned and you could check if a thread with a particular ID was active. A hurry-up rule could spawn a thread with that known thread ID and the thread would simply decrement a counter and sleep, until the counter reached zero. Whatever code ran when you shot a ramp, for example, could check if the thread was alive. If it was, it would call whatever code handled the hurry-up being collected and that code could kill off the timer thread. There was no need for some other flag or variable to hold the state of the rule. Tom felt it was important to replicate this paradigm in Pinball 2000 and made an equivalent framework for “Apple-like” threads. I didn’t need to care about any of that for my work, but there was definitely a lengthy discussion about whether it was ok for these particular threads to be pre-emptive, or co-operative at least among each other.

    The reason I bring this up is that it illustrates something that elevates things from good to great. Everyone on the programming team at Williams was super talented, but we all had strengths and weaknesses. For example, in my opinion and this is only my opinion, the most technically proficient programmers were Lyman, Tom and Duncan and the most creatively proficient were Lyman, Dwight and Keith. Lyman was unique in that he could do it all (I learned more from working with him than anyone else. At his remembrance someone called him “the Clint Eastwood of pinball” and I think that’s a good analogy. I miss him). That diversity of skills was important and it’s why Williams was able to make such good games. We needed our code to be usable by all of the programmers whatever their biggest strength or weakness was. A good piece of code is efficient and flexible and well-designed, but it can require a lot of skill to use well. A great piece of code is all these things except that you don’t need as much skill. It’s intuitive and straightforward and when you make something that uses that code it’s less likely to have bugs. The WPC way was great. The Pinball 2000 way was good. As more game code got written the skill floor for people who’d previously worked with WPC became more apparent. One day this came to a head. I wasn’t present for the discussion but I clearly remember Tom trenchantly saying “I’m making all the Apple-like processes co-operative” and at that point we moved closer to great. There were other cases of this involving my own systems but I don’t remember the details, otherwise I’d use one. Please don’t ever think I’m disparaging any of my colleagues.

« Previous Entries   

Recent Comments