Gamepad.Coder@gmail.com

Made back in 2013. My first large experiment.
Windows can be dragged, zoomed, grouped,
typed into, saved and loaded.

Built in the Unity game development environment, and written in "UnityScript" (a language loosely based on JavaScript). I later ported it to C# in 2014 in order to have better support for polymorphism and subtyping.

My initial plan was to build a framework for AR and VR (Augmented Reality and Virtual Reality), but integrating further support for web, video, audio, and other media seemed like it was going to be a laborious task. So I decided to move on and build on top of platforms where I could easily work with existing files and code libraries, instead of writing each media utility from the ground up.

It was an incredibly fun project, and it was my first big trial by fire where I learned how to better manage and organize large complex projects.

In my first iteration, I had windows in the same plane (parallel to the screen, all the same distance away from the camera). This caused severe visual glitches whenever portions of two or more windows overlapped. I was new to Unity and game engines and 3D, and if there was a workaround for this, I was unable to discover it at the time. To remedy this, I ended up learning a lot about 3D vector math.

[~] note to self, add diagrams here.

I spaced windows away from the others, giving each its own plane in 3D. Then they could move around the user's screen without any anomalies nor visual chimera glitches. But placing windows further from the camera makes them smaller. So as windows are placed further away, they are proportionally resized to maintain the appearance that all of the windows were located in the same "Desktop" plane.

When I move back to working on 3D interfaces again, I may revisit this project, clean it with the better practices I now know, and upload a version of source code which can run on modern versions of Unity. But likely I will instead make a more elegant solution than this one. It was an incredible learning experience though.

Each letter is a node, and each line is a node containing zero or more letters.

Letters contain internal data for their foreground and background color, their font name, their font size, and their style.

The window itself currently has its graphical style hardcoded, but I plan to extend this (much later).

Clicking within the window will activate it, and place the text cursor at the mouse position. Clicking and dragging the mouse will select a region of text. (You can also select text by holding SHIFT and pressing: the arrow keys, HOME button, and the END button.)

When CTRL is held down before the user clicks the window, the window functions differently:

• Holding CTRL and left-click dragging will move the window.
• Holding CTRL and right-click dragging will resize the window.
  (The direction of the resize depends on the quadrant of the window clicked. If the user holds CTRL, right-clicks on the leftmost side of the window, then drags out, the rightmost side of the window will stay where it is, and the leftmost side will follow the mouse and expand or contract.)

Any action which changes the document is called through a COMMANDS class, which creates an Action object to track changes.

After an Action object is created, its Action.do() function is called, then the Action object is pushed onto an undo stack. When the user presses the hotkey for undo (CTRL+Z), the Action object is popped, its Action.undo() function is called, and the Action object is pushed onto the redo stack.

Written in C, then ported to C++. Runs in a Linux terminal, and plots individual pixels to the screen. Keyboard input only.

Windows can be moved and resized. Windows contain lines and headers (which contain sub-sections of more headers and lines). Headers can be folded and expanded, allowing quick navigation between dense sets of hierarchical data without ever taking your hands off the keyboard.

In this app the screen is rendered from blocks, like a chess board. The screen's grid of blocks does not change location nor size, but behind it is a canvas (which can move around) filled with windows and lines and characters (which can also move around). Think of the screen's grid like a camera fixed in place while people move around within (or outside of) its view. Every time the program paints pixels to the screen, the screen's grid "camera" takes a snapshot of blocks it can currently see, and displays them at actual pixel coordinates on the computer's monitor.

The canvas can place windows anywhere, and can scroll to view windows placed off the screen. Windows can be moved and resized. Windows have a solid background color for each block they display within the grid system. Windows can additionally assign a default foreground color for text (different from the workspace's default font color). Windows contain a border of solid-colored grid elements, and a central document. The central document contains rows of lines. Lines can be one of two types: [1] a simple line of character objects to display and edit [2] or a header, which functions as a simple line, but additionally has a sub-document grouped under it. Headers can fold (i.e. collapse and hide their sub-document region, showing only their title) and unfold. Header titles can be edited just like any other line. Sub-documents under headers can contain lines of either type.

When the user selects a window, then activates it, the program shifts into line selection mode. Once the desired line is selected, the user hits [CONTROL+ENTER] to shift from line selection mode -> into editing mode. From here line characters can be typed, selected, copied, pasted, and deleted as if typing in a standard text editor. [CONTROL+ENTER] likewise shifts back from text editing mode -> back to line selection mode.

Lines store a linked-list of character objects. Each character object has: a foreground color, a background color, references to neighbors, and the symbol to display at its grid coordinate. (To specifically differentiate, let's define "symbol" in this context to mean: a letter, number, or any visible character which results from hitting a key on a keyboard.)

When the program draws to the screen, each grid element with a window will report what is located at each blocky grid coordinate. Borders for windows tell the grid to paint a single color at each pixel in that rectangle. Coordinates in a window which do contain a character object report their block's background color and which foreground color the grid should paint their symbol. When the character object tells the grid which symbol to draw, the grid looks up that symbol in the font object.

The font object is a series of 2D regions I programmed manually. Each region has elements containing 1's and 0's representing pixels in a rectangle. The 1's are the pixel positions in the rectangle to draw the symbol, and the 0's represent where to draw the background color. Each typeable character on the keyboard is mapped to one of the 2D regions in the font array (fun example: the 2D region mapped to the space bar is all 0's, resulting in an empty box; pure background color).

So the grid gets the foreground, the background, and the symbol data, then paints the pixels onto the portion of the screen associated with that grid coordinate's region.

This process is repeated for each window and each line on the screen until... viola! You have a screen with blocks and letters and colors.

Next up, how to use it.

• Primary design rule for this application:
  
  You should never have to take your hands off the keyboard nor move them out of typing position (i.e. you'll never have to take your index fingers off the [f] and [j] keys).
  
  

  The [ESDF keys] scroll the canvas,
  which moves all the windows in unison.
  • This lets you scroll around and view other windows.
  • The [A] and [Q] keys adjust the speed at which the canvas scrolls.
  • These controls are only deactivated when in typing mode. Otherwise they work no matter where you are in selection mode.
• The keys are at the core of interaction.
  • The [IJKL] keys are for selection
    across different contexts.
  • The [P] and [;] keys change the scope of what is selected (from high up canvas, to low down character editing).
  • The [U] and [O] keys do a few different things at different levels, but it's pretty straightforward.
• Let's see how keys work at each level
  • When the canvas is active
    • [IJKL] keys select windows.
      
      (The canvas will automatically scroll to view any selected window if off-screen).
    • The [;] key activates the selected window.
    • The [P] key exits an active window and returns the scope to choosing windows within the canvas.
    • The [U] key toggles "move the selected window" mode. Once on, the [IJKL] keys nudge the window (pretend these are the arrow keys). And pressing [U] again turns off move-mode.
    • The [O] key toggles "resize the selected window" mode. [IJKL] keys resize the window. [I] and [K] shrink and expand the window vertically. [J] and [L] shrink and expand the window horizontally. Pressing [O] again turns off resize-mode.
  • When a window is active
    • the [I] and [k] keys allow you to select lines of text
    • the [J] and [L] keys fold and unfold headers
      
      (if the current header is selected and folded then [J] will select and scroll up to the header's parent).
    • the [U] and [O] keys go to
      the next and previous header, respectively.
    • the ([) and ['] keys scroll the current window
      
      (similar to how the [ESDF] keys scroll the canvas)
    • the [P] key will deactivate the window and move selection up to the canvas level.
    • the [;] key or [CTRL+ENTER] combo will activate the selected line (or header) and the program shifts into text editing mode.
  • When a line is active in text editing mode
    • All keyboard inputs which normally type a character will insert a character at the highlighted position.
    • The [ENTER] key will make a new line and move the highlighted position to it. The window is scrolled down one line if the user presses [ENTER] at the bottom of the window's viewable area.
    • [BACKSPACE], [DELETE], [HOME], [END], [PAGEUP], [PAGEDOWN], and the up down left right [ARROW_KEYS] function as they do in regular text editors, such as Notepad.
    • The shift key allows selecting a range of characters in a line (or lines).
    • [CTRL+C] will copy characters if there is a selection.
    • [CTRL+V] will paste characters (replacing any selected characters).
    • [CTRL+ENTER] will exit editing mode and return to line selection mode.

Nano is a minimalistic but fantastic command-line interface text editor that can be found on most Linux distributions.

It uses a code library called ncurses, which I initially used to build the first versions of this project. Ncurses basically lets you define text and colors at positions on a grid, and it makes writing a text editor from (mostly scratch) a breeze.

Later I was curious about how Linux displays images, simple graphics, and command line terminals. So I began investigating the framebuffer.

In addition to using the framebuffer, I retrieved the hardware signals of my keyboard from Linux in the form of scancodes. From there I set up branches of code to account for how to react to different key presses and different combinations.

This experiment was incredibly fun, and was my first foray into interacting with hardware on a low level. Surprisingly, the biggest utility from this project was how it made me start thinking about hardware and software in a more comprehensive way. It really bridged a gap for me between my understandings of high level frameworks <-(&)-> the 1's and 0's that power, organize, and encode everything above. And seeing a bit beneath the hood closer to the machine processes has given me a lasting appreciation for the history of computing in general, and for all the sub-processes that have been working inside my computers all my life.