While shopping at Costo last night, noticed an old friend Robert Green and his beautiful wife Collette happen to be checking out in front of us (they even came down and visited us while in Australia). While chatting the topic of work came up and the fact I am now on the Power BI team and Robert asked if we would be slowing down the crazy/torrid development pace of delivering new features. Not only will we NOT be slowing down a pace of delivery; with the work around Cortana, Azure and continued push to get new collaboration features into the Power BI Service it is likely it will be increasing! With this news Robert asked if i would be interested in doing a Visual Studio Toolbox episodeâ€¦doh -OF COURSE! Since this is developer focused series i decided to show off some of the Power BI features i thought developers would be interested in:
Calling REST APIs, working with JSON and integrating with your Web Development using Power BI
In this weekâ€™s Visual Studio Toolbox episode, old time friend, Charles Sterling joins us to show how to use Power BI in your development efforts specifically how to call REST APIs with Power BI without writing any code. Parsing, modeling and transforming the resulting JSON to make creating rich interactive reports a snap and integrating this into your development efforts by embedding the Power BI data visualizations into your web applications.
Neal Analytics, PowerPivotPro and the Seattle User Group are teaming up to offer free Power BI Training Sept 21st.
Dashboard in an Hour
DIAH is designed to accelerate your Power BI experience with a comprehensive training program all in a single hour. All you have to do is bring your Windows-based Notebook and weâ€™ll supply the rest â€“ even some snacks!
With DIAH you get practical hands-on experience. During this training session attendees will learn:
1. Get Data/Modeling by Avi Singh
2. Visualizing by Charles Sterling
3. Collaborating and Sharing your data by David Brown
Register now to start learning about building great Dashboards.
When: Wed, Sep 21 2016| 7:00PM – 8:00PM PST
Where: Microsoft Building 35
3940 159th Ave NE, Redmond, WA 98052, Redmond, WA (map)
About the Presenters:
Avi Singh is a Power BI trainer and consultant based out of Seattle. He is a Microsoft MVP, co-author of the top-selling book â€śPower Pivot and Power BI: An Excel Userâ€™s Guideâ€ť and a regular speaker at conferences and user events.
Avi has personally experienced the transformation and empowerment that Power BI can bring, going from an Excel user to building large-scale Power BI solutions. His mission now is to share his knowledge about Power BI.
Through his experience with start-ups and with building solutions for Microsoft Consulting Services, David Brown brings expertise in architecting and planning Business Intelligence, predictive analytics, and data warehousing solutions. As Power BI MVP David works in the Managed Partner reporting space with both Microsoft Enterprise and Midmarket teams, creating tools to maximize Microsoft and Partner revenue. After gaining extensive experience with Business Intelligence there, David moved to Microsoft Consulting Services where he worked with the Premier Mission Critical offerings team designing Microsoftâ€™s highest level of support solutions. Davidâ€™s background with Microsoft and his start-up mentality allow him to present the highest quality solutions utilizing Microsoft products and partnerships.
See more at: http://www.nealanalytics.com/our-team/
About Neal Analytics
Neal Analytics is Microsoftâ€™s premier Azure Machine Learning partner, and a data consulting firm with a management consulting approach. We were founded in 2011 out of The Arnold Group, a prominent Management Consulting Firm in the Seattle area, to resolve the need to back consultative findings with analytics. Our management consulting roots have led to a unique approach towards helping customers extract maximal value from their data. When we begin conversations with our customers, we help them see how they can better extract value from their data. We help customers all across the spectrum, from those in need of their first data warehouse for better data management, to those focused on solving industry-leading predictive analytics problems. Our firm is deeply versed in Microsoftâ€™s Cortana Analytics & IoT technology stacks, and we use these platforms to quickly and effectively solve our customerâ€™s problems. Our deep technical competency has led to an industry-leading position in advanced analytics for Retail, Manufacturing, Consumer Goods, Oil & Gas, Tech, and Education as a Microsoft partner. We engage regularly outside of these core competencies in efforts to help customers revolutionize how they do business with data. If you want to discover how to extract more value from and drive real improvement in your business with data, you should contact Neal Analytics.
It all starts with a single observation: truly impactful analytics are only possible when â€śthe businessâ€ť is directly involved, in a â€śhands-onâ€ť capacity. The alternative â€“ communicating requirements to IT or consultants, awaiting results, and then iterating â€“ takes too long, costs too much, and delivers too little. If you have ever wondered why Business Intelligence projects seemingly run forever while delivering poor ROI, or why spreadsheets continue to dominate the landscape of reporting and analysis, the answer to both lies in â€ścommunication costâ€ť â€“ either paying its enormous cost (in the case of BI projects) or avoiding it (by the Biz taking matters in their own hands, with Excel).
Said another way, your Business cannot merely Consume analytics built by someone else â€“ it must take an active and ongoing role in Producing them. Donâ€™t worry though, only 1-2 people in each department need to wear the Producer hat, and in all probability, they are already wearing it (using Excel to produce reports for the rest of their workgroups). You need only to empower those people with the new â€śImproved Excelâ€ť toolset, one that they will gratefully embrace once theyâ€™ve glimpsed its capabilities. And rather than cutting IT out of the equation, this New Way provides the first-ever incentives and foundation for true cooperation between IT and the Biz.
If you are a Power BI customer in the Chicago area looking for Power BI training, join Rightpoint for some of the best Power BI training available, an end to end look at Power BI with the Dashboard in a Day (DIAD) training jointly developed with Microsoft.
Chicago Dashboard in a Day
DIAD is designed to accelerate your Power BI experience with a comprehensive training program in a single day. All you have to do is bring your Windows-based Notebook and weâ€™ll supply the rest â€“ even Lunch!
With DIAD you get practical hands-on experience. During this training session attendees will learn:
- How to connect to, import & transform data from a variety of sources
- Build real data models, as well as author and publish Business Intelligence reports
- Customize and share your â€ścreationsâ€ť for collaboration with other groups securely inside your organization including mobile device sharing
Register now to start learning about building great Dashboards.
Wed, Sep 21 2016| 8:00AM – 5:00PM CST
200 East Randolph Drive, Suite 200
Chicago, IL 60601
About the Presenter:
SOLUTION SPECIALIST, ENTERPRISE
Neal is responsible for helping our clients create a vision for the “Connected Company.”
Neal takes the time to understand specific client needs and translates those needs to specific solutions for enterprise collaboration and insight.
Neal has extensive experience in enterprise collaboration and business intelligence. Before joining Rightpoint, Neal spent seven years at Microsoft Corporation in a variety of Pre-Sales technical roles. Prior to joining Microsoft, Neal was enterprise solutions consultant with KPMG Consulting/BearingPoint, focused on CRM and ERP solutions.
Rightpointâ€™s business philosophy is to rethink the “typical” consulting model by combining attributes of management consulting, IT consulting and creative agency services to help clients drive business results by solving strategic problems. Rightpoint’s pervasive “intrapreneurial” spirit, which ensures that clients “get it right” the first time, is the foundation by which we have built a strong, national reputation for excellence.
I talked about the different aspects of building a database engine in detail in the past month or so. But I tried to talk about each topic independently, so it will make sense. The problem is that in the real world, there are actually quite a lot of related stuff that impact on one another. This series of posts is meant to tie everything together, so youâ€™ll have a better understanding how the design decisions in one place being affected by the requirement in somewhere that seems utterly unrelated.
Before we can talk about the implementation details, let us see what we are trying to achieve. Voron is:
- High performance.
- Single write, multiple readers (MVCC)
- Fully ACID
In this post, Iâ€™m not going to talk about the data model, or how we sort it, or anything like that. No, we are at a much lower level than that. We are at how we access the raw data pages and manage them.
There are actually multiple players involved here. We have the journal for durability of writes, we have the data file to store the data, the scratch file to implement Multi Versioning Concurrency Control and the Page Translation Tables to provide snapshot isolation for concurrent transactions.
The design of Voron is immensely simplified by the fact that we choose to go with a single writer model. We share this design decision with other databases engines such as LMDB, LevelDB, RocksDB, etc. Concurrent write transactions are much more complex and require a lot more effort, and you still have the serialization at the journal level, although I explored multiple ways around it. With Voron, we decided to go with a single write transaction for the simplicity, and then implemented transaction merging on top of that, which give us a tremendous performance boost in high load scenarios.
But let us talk about MVCC. The idea is that we have concurrent versions of the data, so each transaction has a snapshot of the entire database and can operate on that without fear of write transactions modifying data while it is running. Let us explore how this works when the database starts.
The key to that is the notion of the page translation table, from now on, known as the PTT. When the database starts, we have an empty PTT, and the data file itself. We open a read transaction, which has the following data:
- PTT: [ /* no entries */
- Data file
Whenever the read transaction need to read a page, it consults the PTT, find that there is nothing there, and read the page from the data file. We keep the read transaction open, and open a new write transaction. It also gets a PTT and the data file, but it also needs to keep track of a few other things:
- PTT: [/* no entries */]
- Data file
- Dirty pages
Now, we want to make a change to the database, which happens to fall on Page #3. Here we have problem, we canâ€™t modify the data file directly, ReadTx-1 is still running, and it might want to read the data in Page #3 at some point. Instead of modifying the data directly, we copy the page into the scratch file.
The scratch file is just a temporary file that we use to store data copies. After we copy the data, we update the PTT. Now when we search for Page #3, weâ€™ll find the location of the page in the scratch file. As far as the write transaction is concerned, this doesnâ€™t matter. A page is a page is a page, and it doesnâ€™t matter where it is at.
Committing the transaction means taking all of the dirty pages in the write transaction and writing them to the log. After which we atomically set the PTT for the write transaction as the global PTT. Now, all future read transactions will have the new PTT and when they will ask for Page #3, they will get the page from the scratch file.
A new write transaction that needs to (again) modify Page #3, will create another copy of the Page inside the scratch file.This ends up looking like this:
We have three copies of Page #3. One for the original read transaction (in the data file), one for the current read transactions (yellow in the scratch file) and the current modified page (orange in the scratch file) that we are writing to.
When the write transaction completes, we again flush the dirty pages to the journal and then publish our PTT so all future transactions can see the changes.
Of course, that is just one side of it, in my next post, Iâ€™ll discuss how we clear the scratch file and move data back to the data file.