The Design of Embedded Systems for Outdoor Augmented Reality (AR) Applications Integrated to See-Through Glasses – D.Rama Krishna, K.Sharanya & P.Swathi

November 28, 2015 Yuva Engineers Uncategorized 0

Title: The Design of Embedded Systems for Outdoor Augmented Reality (AR) Applications Integrated to See-Through Glasses

Authors: D.Rama Krishna, K.Sharanya & P. Swathi

Organisation: Malla Reddy Engineering College for Women

Abstract- The arrangement of undertakings incorporates article situating, realistic calculation, and in addition remote interchanges, and we consider requirements, for example, continuous, low power, and low foot shaped impression. We present an unique mariner help application, as a normal, valuable, and complex open air AR application, where  setting subordinate virtual items must be set in the client field of perspective as indicated by head movements and encompassing data. Our study exhibits that it merits taking a shot at force streamlining, following the installed framework in light of a standard universally useful processor (GPP) + illustrations handling unit (GPU) expends more than high-glow transparent glasses. This work introduces then three principle commitments, the first is the decision and mixes of position and state of mind calculations that fit with the application connection. The second one is the building design of the implanted framework, where it is presented as a quick and straightforward item processor (OP) streamlined for the space of versatile AR. At long last, the OP executes another pixel rendering technique (incremental pixel shader (IPS)), which is actualized in equipment and exploits OpenGL ES light model. A GP+OP(s) complete construction modeling is depicted and prototyped on field programmable door exhibit (FPGA). It incorporates equipment/programming parceling in view of the investigation of utilization prerequisites and ergonomics. 

Keywords: Processor Architecture; Communication/memory optimization; 3D graphics; Inertial sensors; 3D positioning.

INTRODUCTION

Recent breakthroughs in augmented reality (AR) display will bring a lot of new applications in the future. However, this also means an emerging challenge, which is the design of low-cost, low-footprint, and low-power systems to be embedded in see-through glasses. On the one hand, most of research works in related conferences (e.g. ISMAR), do not focus on embedded system design but on specific AR issues such as simultaneous localization and mapping or virtual object handling. On the other hand, emerging see-through glasses are considered as a new peripheral connected to smart phones, which can execute AR applications. The contributions of this work are twofold. The first one is an in-depth study of algorithms for the target AR application in the domain of sailor assistance. The second one is the design of an architecture  that fits with application requirements. The application domain is outdoor AR without any equipment in the user environment. Such applications require the management and the fusion, at runtime, of multiple information flows (position/attitude sensors, points of interest, …) and simultaneously the processing of graphical objects. The paper is organized as follows. In Section 2, we present our motivations for this research field and the specificities of the target applications. In Section 3, we present current technologies for AR and a state of the art of usual implementations. Section 5 describes the main steps of our original approach. Our solution is based on the adaptation of previous positioning solutions to the application context, on algorithmic transformations to improve data reuse and reduce processing load, and on a new architectural solution for object drawing. Section 6 presents our hardware/software architecture based on data locality and bandwidth optimization. Our architecture is dedicated but flexible and optimized according to AR applications;

Outdoor AR applications

AR without any other individual’s info is not another point, but instead various troubles stay unsolved. This is especially legitimate in flexible and outside settings, where field markers are not applicable and video-demonstrated based after is difficult to execute, in light of current circumstances, brilliance conditions. The proposed application set relies on upon the going with observations. To begin with, organizers starting now have accessible to them a huge measure of recorded and described data of geo-restriction. Second, these data can be added to a customer’s field of vision by technique for see-however glasses. Third, an embedded system can handle 3D dissents that fit with the scene, if appropriate sensors can give the customer mien. Fourth, we consider an application-based evacuated articles that don’t require an amazingly exact arranging, so no camera is required for the stance estimation. Finally, vital ergonomics rules for arranging applications require direct and noteworthy things in the customer field of point of view. In this association, various applications, in non military work force and military spaces, can be proposed to upgrade security and course decisions with particular without hands and low-foot formed impression devices. In this association, we consider the particular however tangled logical examination of sailor assistance from which can be derived diverse necessities for a nonexclusive introduced system. This relevant examination is plot . On a boat, the perception of the position is vital when moving nearer fragile circumstances, for instance, coasts, immeasurable sea reefs, high-thickness course zones, or harbor channels. These circumstances are honest to goodness for small cruising or speedboats and for generous vessels where the course gathering is obliged with respect to watercraft sizes. In this kind of broad water crafts, it is in like manner recommended to join visual checking, considering bona fide environment recognitions, and instrument guiding. Current schedules include doing an inversion and forward between aide examination and visual recognitions. Regardless, organizing aide signs with a bona fide circumstance can be questionable and oversight slanted. It in like manner identifies with lost time that can be profitable if there ought to be an event of emergency. Finally, this organizing can be basically inconceivable when the deceivability is terrible.

This is a noteworthy logical examination since a watercraft is a particularly unstable structure. All the procedure with developments have diverse parameters depending upon boat speed, customer moves, and ocean movements. Swell periods can move some place around 0.05 and 0.1 Hz. Regardless, this is moreover a space where a huge amount of data are available. The main grouping fuses static seamarks objects. The second one is intense yet can be surveyed, and in this class, we find for event the ocean streams and the tide-ward shallow extents. The third one is given by the modified unmistakable confirmation system (AIS) that demonstrates the positions, the heading, and the ID of barges or any ocean objects in nature. All these data can be added to the customer fields of vision as showed by position and perspective estimations. By then, we have boat arranging data, which join overall arranging structure (GPS) estimations, speed, trim, heel, and heading. All these data, which can be encoded in sensitive or military applications, can be overcome a remote framework that does not require high information transmission limits. Then again, these data are useful yet not sufficient since it is critical to know the customer demeanor, which is portrayed with the head saucy positions. These data must be given by embedded sensors that must be essential with the glasses. Abundance amidst vessel and customer data can in like manner be advantageously joined to improve accuracy. For instance, the on-glasses accelerometers can be combined with the watercraft GPS to gage neighborhood continues ahead a long vessel. The force of the straightforward device is moreover a fundamental restriction in such outside applications. It must be pleasing with the atmosphere conditions, for instance, storm, dimness, or extraordinary light appearance in sunny conditions; moreover, the structure ought to in like manner be usable amid the night. Finally, ergonomics are of the best hugeness in course applications where the objective is to give the customer profitable and critical information. Inputs from potential customers we are working with, in both non military faculty and military spaces, assert that the amount of articles must be obliged and that question structures must be clear. These essentials let us consider that as a representation taking care of unit (GPU) is truly excessive, it in like manner suggests open entryways for utilization changes.

IMPLEMENTATION

AR display:-

As presented in Section 2, our class of applications requires optical see-through displays 2. Considering weather conditions (light reflection, fog, humidity), it turned out that Laster was providing the better solution for our outdoor conditions. The maximum light intensity of the product (ski mask) is about 5,000 cd/m2 in color mode and monochrome is even higher. The resolution and the field of view are suitable for a see-through device: 800 × 600 and 40°×30°, respectively, which is better for light intensity (×10) and field of view (×2) than other solutions. The display technology is based on a LCD panel with LED backlight to reach expected high luminosity that cannot yet be delivered with AMOLED technologies.

SoC and development board:-

We have implemented a first version of the application on a ST-Ericsson Snowball board, which includes the A9500 SoC (see Figure 1). It is a GPP+GPU architecture that implements a double cortex A9 with a flexible clock frequency between 200 MHz and 1.0 GHz and a GPU Mali-400 MP. It also includes a wireless communication chip that provides Bluetooth (BT), WiFi, and GPS. Head tracking can be performed with the MEMS available on the board, which are an accelerometer/magnetometer 3-axis (LSM303DLHC from ST) and a gyroscope 3-axis (L3G4200D from ST). The GPS data are supposed to be available from the boat or user throughout the lowbandwidth wireless (BT) connection, so the GPS and the WIFI are switched off.

A9500 application processor ST-Ericsson

Figure 1 A9500 application processor ST-Ericsson.

OS and software:-

The mobile OS is Android; it offers a large amount of application programming interfaces (APIs) that simplify GPU programming with OpenGL ES 2.0 API [12] and allows to control and acquire MEMS data (sensor Manager) to get position data from a GPS (Location Listener) and some managers to handle wireless connectivity (BT, WiFi). As a proof of portability, we have also implemented the application on a Galaxy S3 Smartphone, which is based on the same GPU. In AR applications, the main functionality is the graphic service and the system has to draw 2D or 3D objects with different colors and textures. The second service is the attitude computation or head tracking, and this step is needed to place right objects in right orientation on the display. A special task with MEMS is implemented and it is mainly based on a standard 7- states extended Kalman filter [13]; we come back on this type of filter in Section 5. The MEMS data rate is set to 50 Hz. Finally, a wireless service was developed to acquire data from the boat network (GPS, AIS, map sea-marks, wind and stream grids) when those data are available from a PC on the boat. We used the NMEA protocol commonly used in marine electronics and BT for wireless communications, that do not require high bandwidth.

Power measurements:-

We measured the power consumption of the Snowball board and the Lester mask with different configurations, and the results are given in Table 1. The mask has five levels of luminosity, and the highest one consumes 130 mA. We then observe that the power consumption of the embedded system is much larger than the AR display, and it consumes 310 mA with a basic kernel and reaches 500 mA with Android, BT, MEMS (without gyroscope), and the application running with a set of simple objects. Expressed in watts, the SB board consumed 2.5 times more power than the mask with the highest luminosity. We note that the Bluetooth connection (version 2.1-EDR) significantly increases the power consumption (110 mA), in practice, this overhead can be drastically reduced with a low-power version (e.g., 4.0).

RESULTS

In this segment, we give and examine results identified with execution estimations, data transmission use, usage cost on FPGA, and force utilization.

Exhibitions First, the entire application stream has been portrayed with a parameterized execution demonstrate that empowers the tallying of operations and exchanges as per 25 variables, for example, the showcase arrange, the number and size of items, and so on. At that point, this model has been accepted with some genuine profiling completed on the objective MB with and without floating point unit (FPU). The execution model of the equipment usage is direct following the quantity of ventures of the FSM is completely indicated.

The results with the following configurations:

– ‘SW’: fully software implementation on a MB.

– ‘SW+FPU’: MB with a floating-point unit.

 – ‘SW+FPU+OP IP’: MB with FPU and one OP unit.

In this illustration, we consider a case, which goes past our contextual investigation necessities. Therefore, in this setup, we consider 8 objects, 16 vertices, and 13 polygons for every item, a VGA show, a normal 2D article size equivalent to 1/20 VGA and a sensor (IMU MEMS) obtaining rate of 50 Hz while the GPS securing rate is 5 Hz. On the off chance that we consider a 100 MHz clock (100.106 cycles), we watch that the situating/mentality control a portion of the application requires 70% of accessible cycles with the SW arrangement. The edge is too little to be in any way safe inside of a Linux framework where extra client procedures are vital. Also, a FPU unit is of genuine quality for network operations that have high exactness prerequisites. The second arrangement “SW+FPU” can likewise be considered for the article situating (steps 1.1, 1.2, 1.3) that requires around 2 M of cycles, however the quaternion-based pivot of items will be excessively covetous (66 M cycles) and would prompt an aggregate of 80% of CPU use. Of course, the realistic piece of the application is unquestionably out of the extent of any processor-based usage, aside from the 2D projection steps (3.8 M). In any case, joined with past application necessities, this operation can’t be mapped on the processor and the change would not bode well as far as information area. This implies the OP HW IP will handle all the realistic strides with a reasonable interface to the product situating part that can sustain the OP with sifted state of mind and position information. The OP processor is then accountable for adjusting item attracting as indicated by new information. Figure 13 gives the advancement of exhibitions for the three arrangements. They are straight with the number and the measure of articles, and the size is given as a proportion of VGA casing. We watch that the exhibitions of the OP usage stay three requests of size superior to anything SW ones. These components can likewise be utilized to choose the quantity of OP to be executed if vital. Taking into account the past suppositions, 40 M cycles are required for a complete execution of the representation part. Tables 2 and 3 give the usage aftereffects of the OP HW IP on diverse Xilinx gadget families: Virtex 5 and 6 and Spartan 6. The last one displays the least clock recurrence (73 MHz), which required to fully implement the Y-buffer on chip, such a chip can implement four OPs. With the next generation of Virtex 6, we note that the smallest device (LX75) is large enough to design a system with four OPs with an on-chip Y-buffer.

On-chip memory size

Figure : On-chip memory size vs number of objects (seven polyg./Obj) and polygons (three objects).

A LX 130 can theoretically implement 48 OPS but is limited to 12 according to the memory bandwidth capacities. Virtex are expensive FPGAs while the Spartan family has been designed for low-cost design that fits the context of embedded systems. We observe that a small LX25 Spartan 6 can implement one OP and a LX45 provides enough resources for three OPs. The first solution that allows for an in-chip Y-buffer is the largest Spartan that can implement 12 OP according to bandwidth constraints. So low-cost solutions with high performances are possible and more can be expected. Xilinx has released in 2011 a new device generation that relies on a 28-nm technology and brings significant power savings. This is especially true with the low-cost, low-power Artix 7 family that opens interesting perspectives. The positioning/control part of the application can be mapped on one A9 core, while two OPs can be synthesized on smallest Zynq device (XC7Z010).

CONCLUSIONS

Few examination work has been directed in the space of installed framework plan for versatile enlarged reality applications in the setting of developing light transparent HMD. In this task, we have directed an inside and out calculation/building design contemplate and planned a complete framework as indicated by solid foot shaped impression limitations. The arrangement that has been created is committed however adaptable. The methodology has purposely been centered around standard conventions and interfaces; it can be interconnected with common inertial sensor and correspondence peripherals. This work results in another methodology for the configuration of AR particular installed and reconfigurable frameworks with three fundamental commitments. The first is the decision and the full determination of a whirligig free arrangement of calculations for position and state of mind estimation, and this arrangement depends on the affiliation and the adjustment, to the AR area, of diverse past commitments. The second one is the installed framework building design, where it is presented as a quick and straightforward article processor (OP) improved for the space of portable AR. The structural engineering is particularly advanced for information reuse and adaptable since articles can be disseminated on a given number of OPs. At last, the OP executes another pixel rendering system (IPS) actualized in equipment and that exploits OpenGL ES light model suggestions. The entire structural planning has been executed on different FPGA targets. The outcomes exhibit that normal exhibitions can be come to and that an ease FPGA can execute numerous OPs. This arrangement is reasonable when reconfigurable architectures bode well, yet a definitive answer for mass business sector items would be a chip including a GPP and various OPs with force gating capacities. Current experiencing work concentrates on the decision of articles to be shown in the client field of perspective. This study is made mutually with ergonomists and taking into account client criticisms. The decision of articles to show will be actualized as a product administration that will be accountable for the neighborhood memory content.

REFERENCES

  1. D Li, RJ Landry, P Lavoie, Low-cost MEMS Sensor-based disposition determination framework by coordination of magnetometers and GPS: a genuine information test and execution assessment. IEEE Position Location and Navigation Symposium, 1190–1198 (2008)
  2. S Nasiri, A basic survey of MEMS whirligigs innovation and commercialization status. Specialized report, InvenSense (2010).
  3. K Mirza, K Sarayeddine, Key difficulties to moderate transparent wearable shows: the missing connection for mobilearmass sending. Specialized report (2012)
  4. AR Lingley, M Ali, Y Liao, R Mirjalili, M Klonner, M Sopanen, S Suihkonen, T Shen, BP Otis, H Lipsanen, BA Parviz, A solitary pixel remote contact lens show. J. Micromechanics Microengineering. 21(12), 125014 (2011)
  5. A Herout, M Zacharias, M Dubska, J Havel, in Mixed and Augmented Reality (ISMAR) 2012 IEEE Int. Symp. On. Fractal marker fields: No more scale constraints for trustee markers, (2012), pp. 285–286
  6. AI Comport, E Marchand, M Pressigout, F Chaumette, Real-time markerless following for enlarged reality: the virtual visual servoing structure. IEEE Trans. Perception Comp. Chart. 12(4) (2006)
  7. J Karlekar, SZ Zhou, W Lu, ZC Loh, Y Nakayama, D Hii, in Mixed and Augmented Reality (ISMAR), 2010 ninth IEEE International Symposium On. Situating, following and mapping for open air increase, (2010), pp. 175–184
  8. X Yang, TT K-Cheng, in International Symposium on Mixed and Augmented Reality (ISMAR). LDB: A ultra-quick element for adaptable augmened reality on cell phones, (2012)
  9. S Feiner, B MacIntyre, T Hollerer, A Webster, in Wearable Computers, 1997. Summary of Papers., First International Symposium On. A visiting machine: prototyping 3D portable enlarged reality frameworks for investigating the urban environment, (1997), pp. 74–81
  10. Mama Livingston, LJ Rosenblum, SJ Julier, D Brown, Y Baillot, J Edward, S Ii, JL Gabbard, D Hix, In Interservice/Industry Training, Simulation, and Education Conference. An expanded reality framework for military, (2002), p. 89
  11. J Kingyens, JG Steffan, The potential for a GPU-like overlay structural engineering for FPGAs. Int. J. Reconfigurable Comput. (2011). doi:10.1155/2011/514581
  12. A Munshi, D Ginsburg, D Shreiner, OpenGL(R) ES 2.0 Programming Guide. (Addison-Wesley, Boston, 2008) 13. JL Marins, X Yun, ER Bachmann, RB Mcghee, MJ Zyda, in Intelligent Robots and Systems, 2001. Procedures 2001 IEEE/RSJ International Conference On. A developed Kalman channel for quaternion-based introduction estimation utilizing marg sensors, vol. 4, (2001), pp. 2003–20114
  13. A Waegli, J Skaloud, P Tome, J-M Bonnaz, in ION-GNSS 2007. Appraisal of the incorporation methodology in the middle of GPS and body-worn MEMS sensors with application to brandishes, (2007)
  14. KH Kim, JG Lee, CG Park, Adaptive two-stage amplified Kalman channel for a flaw tolerant INS-GPS inexactly coupled framework. Aviation Electron. Syst. IEEE Trans. 45(1), 125–137 (2009)
  15. E-H Shin, N El-Sheimy, in Position Location and Navigation Symposium, 2004. PLANS 2004. An unscented Kalman channel for in-movement arrangement of minimal effort IMUs, (2004), pp. 273–279
  16. W Koo, S Chun, S Sung, YJ Lee, T Kang, in National Aerospace and Electronics IEEE Conference (NAECON). In-flight heading estimation of strapdown magnetometers utilizing molecule channels, (2009)
  17. D Gebre-Egziabher, GH Elkaim, JD Powell, BW Parkinson, in Position Location and Navigation Symposium, IEEE 2000. A without gyro quaternion-based mentality determination framework suitable for execution utilizing minimal effort sensors, (2000), pp. 185–192 19. J Bijker, W Steyn, Kalman channel arrangements for a minimal effort inexactly coordinated inertial route framework on a carrier. Control Eng. Pract. 16(12), 1509–1518 (2008) 20. P-F Guo, H Qiu, Y Yang, Z Ren, in Position Location and Navigation Symposium (PLANS). The delicate iron and hard iron adjustment technique utilizing broadened Kalman channel for state of mind and heading reference framework, (2008)
  18. J Bresenham, Algorithm for PC control of an advanced plotter. IBM Syst. J. 4(1), 25–30 (1965)
  19. S Wuytack, J-P Diguet, F Catthoor, H De man, Formalized strategy for information reuse investigation for low-control various leveled memory mappings. IEEE, Trans. VLSI Syst. 6(4), 529–537 (1998)
  20. M Franklin, in RTO Human Factors and Medicine Panel (HFM) Workshop. The lessons learned in the use of increased reality (NATO West Point, NY, USA, 2006)
  21. R Vaslin, G Gogniat, J-P Diguet, E Wanderley, R Tessier, W Burleson, A security approach for off-chip memory in installed microchip frameworks. Microprocess. Microsyst. 33(1), 37–45 (2009)
  22. D Suzuki, M Natsui, A Mochizuki, S Miura, H Honjo, K Kinoshita, H Sato, S Ikeda, T Endoh, H Ohno, T Hanyu, Fabrication of an attractive passage intersection based 240-tile nonvolatile field-programmable entryway cluster chip skipping squandered compose operations for insatiable force diminished rationale applications. IEICE Electron.

To download the above paper in PDF format Click on below Link:

 DRamaKrishna-KSharanya-PSwathi