# Project: Improving FHIR Analytics Using Apache Spark and Cassandra **URL:** \ **Category:** Google Summer of Code\ **Tags:** fhir, toolkit, gsoc2019-project, gsoc2019\ **Created:** [January 27, 2019, 5:38am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845 "2019-01-27T05:38:35Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![prashadi](https://forums.librehealth.io/user_avatar/forums.librehealth.io/prashadi/32/460_2.png) [@prashadi](https://forums.librehealth.io/u/prashadi)\ **Post date:** [January 27, 2019, 5:38am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/1 "2019-01-27T05:38:36Z")
Hi @sunbiz, @namratanehete, @r0bby, During the last year GSoC we have built the base of the [FHIR analytics module](https://forums.librehealth.io/t/project-fhir-analytics-using-apache-spark-and-cassandra/2411), now we can focus on integrating the module to the platform and improve the features of it. The current Librehelath FHIR analytics module is powered by Apache Spark, Apache Cassandra and Cerner Bunsen modules with Spring framework. This module provide flexibility to analyze FHIR data which stored in the Cassandra Database. Module utilize the functionality provided by the Cerner Bunsen module with Spark SQL which provide convenient interface to query FHIR data using different parameters. Even this module allow users to join multiple FHIR resources and find relationships through Spark SQL query language. The repository can be found [here](https://gitlab.com/librehealth/toolkit/lh-toolkit-fhir-analytics). # Objectives - Integrate the FHIR analytics module with the platform - Upgrading Cerner Bunsen Module to get the latest functionality - Integrate with Apache Drill - Enhance query interface and incorporate with FHIR query builder # Prerequisites - Non trivial contribution for Librehealth platform - Good understanding of Spring, FHIR, Cassandra and Spark SQL - Try out the module functionality # Extra Credit - Write a paper on the accomplished work during the project ---
**Author:** ![mshubhankar](https://forums.librehealth.io/user_avatar/forums.librehealth.io/mshubhankar/32/982_2.png) [@mshubhankar](https://forums.librehealth.io/u/mshubhankar)\ **Post date:** [February 20, 2019, 6:34pm UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/14 "2019-02-20T18:34:57Z")
Hey @prashadi, I am interested in this project and would be very glad to know more about it. I see that there are no issues to work on currently. I have built the repo from Gitlab and would be interested in knowing how to start contributing more from you. ---
**Author:** ![prashadi](https://forums.librehealth.io/user_avatar/forums.librehealth.io/prashadi/32/460_2.png) [@prashadi](https://forums.librehealth.io/u/prashadi)\ **Post date:** [February 21, 2019, 8:18am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/15 "2019-02-21T08:18:10Z")
Thank you for your interest @mshubhankar I’ll check and add few issues. But meantime it’s not must to do a contribution for this project only. You can do a contribution for existing projects such as toolkit and radialogy. ---
**Author:** ![mshubhankar](https://forums.librehealth.io/user_avatar/forums.librehealth.io/mshubhankar/32/982_2.png) [@mshubhankar](https://forums.librehealth.io/u/mshubhankar)\ **Post date:** [February 21, 2019, 4:07pm UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/16 "2019-02-21T16:07:07Z")
Sure. I have been trying to solve issues from lh-ehr project but would also like to get some hands on experience in this project and exactly understand how it is going to be. ---
**Author:** ![prashadi](https://forums.librehealth.io/user_avatar/forums.librehealth.io/prashadi/32/460_2.png) [@prashadi](https://forums.librehealth.io/u/prashadi)\ **Post date:** [February 22, 2019, 4:54am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/17 "2019-02-22T04:54:07Z")
@manushree That’s great. I’m hoping to add set of issues soon. ---
**Author:** ![MisterAwesome23](https://forums.librehealth.io/user_avatar/forums.librehealth.io/misterawesome23/32/954_2.png) [@MisterAwesome23](https://forums.librehealth.io/u/MisterAwesome23)\ **Post date:** [March 1, 2019, 8:17pm UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/18 "2019-03-01T20:17:01Z")
Hello. My name is Manthan Admane [UserID: MisterAwesome] By looking at an overall abstract, I am definitely interested in contributing and pushing this project positive steps forward. At the same time being very honest most of the terms [“Apache Spark”, " Apache Cassandra", “Cerner Bunsen modules”] in project description are daunting to me, infact to be equally honest I’ve heard of them for the first time. XD. Present condition is, I am pursuing my B.Tech in Computer Science (Junior/Third Year) and aware of related terms like MYSQL DB, Apache servers, etc. I am interested in web technology hence have a basic competency in that section too. I do feel confident (and hope) that I will catch onto all the technical requisites for this project by putting in the required efforts. By the way I am already on with Non trivial contribution for Librehealth platform with lh-ehr : ) Looking forward to this positively. Thank you. P.S: I am already on research and to know more of all the new terms and technologies ; ) ---
**Author:** ![mshubhankar](https://forums.librehealth.io/user_avatar/forums.librehealth.io/mshubhankar/32/982_2.png) [@mshubhankar](https://forums.librehealth.io/u/mshubhankar)\ **Post date:** [March 10, 2019, 4:38pm UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/19 "2019-03-10T16:38:56Z")
Hey @prashadi ! Haven’t heard from you in a long time. How far are we with the new issues ? I have been through your blogs and I would like really like to know more about where you have stopped and where I can begin contributing. As I see it, bunsen has been upgraded to 0.4 after where you have stopped and so has spark to 2.3 and hapi to 3.6 . After we are done with that, we can add functionality for Drill and enhance the query interface (need to know more about that as well), which should be the realtively harder part of the project. ---
**Author:** ![prashadi](https://forums.librehealth.io/user_avatar/forums.librehealth.io/prashadi/32/460_2.png) [@prashadi](https://forums.librehealth.io/u/prashadi)\ **Post date:** [March 11, 2019, 3:35pm UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/20 "2019-03-11T15:35:57Z")
@mshubhankar sorry for the delay. Saw you have sent a pull request to the project. Were you able to test the functionality after the upgrade? Pull request is looking good to me. ---
**Author:** ![mshubhankar](https://forums.librehealth.io/user_avatar/forums.librehealth.io/mshubhankar/32/982_2.png) [@mshubhankar](https://forums.librehealth.io/u/mshubhankar)\ **Post date:** [March 12, 2019, 12:14am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/21 "2019-03-12T00:14:53Z")
@prashadi Yeah there’s no problem in creating the war file and deploying it on the server. But there has been a weird problem lately. It is also happening without the changes that I have made. The sql query is timing out and the app gets stuck on “please wait..” . I am attaching an image for your reference. ![23%20PM](https://forums.librehealth.io/uploads/default/original/1X/605564dc49201852bd3d9c26d8ce96bff9222d02.png) Note that this is happening also without the changes in PR and this wasn’t happening before. ---
**Author:** ![prashadi](https://forums.librehealth.io/user_avatar/forums.librehealth.io/prashadi/32/460_2.png) [@prashadi](https://forums.librehealth.io/u/prashadi)\ **Post date:** [March 12, 2019, 5:04am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/22 "2019-03-12T05:04:32Z")
@mshubhankar can you redeploy the entire application by cleaning the webapps directory? Do you see any backend error logs? ---
**Author:** ![mshubhankar](https://forums.librehealth.io/user_avatar/forums.librehealth.io/mshubhankar/32/982_2.png) [@mshubhankar](https://forums.librehealth.io/u/mshubhankar)\ **Post date:** [March 12, 2019, 3:42pm UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/23 "2019-03-12T15:42:48Z")
@prashadi I found the problem. I have both Java 8 and Java 11 on my laptop and the war file was built on Java 8 whereas tomcat was running on Java 11. Anyway, things are good and up running now. I tested the functionality after the PR and it’s working just fine 🙂 ---
**Author:** ![prashadi](https://forums.librehealth.io/user_avatar/forums.librehealth.io/prashadi/32/460_2.png) [@prashadi](https://forums.librehealth.io/u/prashadi)\ **Post date:** [March 17, 2019, 10:00am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/24 "2019-03-17T10:00:02Z")
Great I’ll merge the pull request after I tested locally as well. Good work. ---
**Author:** ![mshubhankar](https://forums.librehealth.io/user_avatar/forums.librehealth.io/mshubhankar/32/982_2.png) [@mshubhankar](https://forums.librehealth.io/u/mshubhankar)\ **Post date:** [March 25, 2019, 6:20am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/25 "2019-03-25T06:20:48Z")
Hi @prashadi @sunbiz @namratanehete @r0bby, I’m starting to write my proposal and wanted to get my objectives sorted out. I have some questions related to some objectives in mind and would be glad if someone can help me out with these : 1. **Upgrade FHIR analytics to work with latest cerner bunsen -** The latest cerner bunsen depends on Spark 2.3 and Hapi 0.4.6 . I have made these changes and requested a PR with the updates. Another objective of the project is also to get the latest functionality of bunsen but I’m not sure which functionalities we can add to the project. Do we want some added functionalities from [here](https://github.com/cerner/bunsen/blob/master/CHANGELOG.md) ? 2. **Complete the TODO Implementations in utils/LibrehealthAnalyticsUtil.java file** 3. **Integrate with Apache Drill** - Apache drill has a good documentation to be built with JDBC drivers and also in the Spark environment but there seems to be some problem with it being integrated with cassandra. The documentation mentions that a custom plugin can be made for cassandra but I don’t find any prior work to refer to. I have also found a [post](https://confusedcoders.com/data-engineering/apache-drill/sql-on-cassandra-querying-cassandra-via-apache-drill) which is old and deprecated but can be used as reference. It would be great to have some information about this. 4. **Enhance query interface** - Are there any particular interface enhancements ? Have I missed out on something except of writing a research paper which I hope can be done after being done with the objectives ? ---
**Author:** ![mshubhankar](https://forums.librehealth.io/user_avatar/forums.librehealth.io/mshubhankar/32/982_2.png) [@mshubhankar](https://forums.librehealth.io/u/mshubhankar)\ **Post date:** [March 28, 2019, 12:13am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/26 "2019-03-28T00:13:07Z")
Hey @prashadi @sunbiz @namratanehete @r0bby, I’ve written the first draft of my proposal [here](https://docs.google.com/document/d/1BRJdEXm6vpUZWWvg7frCsz-CXPjCz6Mdy_DNDxjbmJE/edit#). Please have a look and revert me back with your feedback and valuable comments to make it better. ---
**Author:** ![r0bby](https://forums.librehealth.io/user_avatar/forums.librehealth.io/r0bby/32/1781_2.png) [@r0bby](https://forums.librehealth.io/u/r0bby)\ **Post date:** [March 28, 2019, 1:45am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/27 "2019-03-28T01:45:14Z")
> [@mshubhankar](#): > > Hey @prashadi @sunbiz @namratanehete @r0bby, > > I’ve written the first draft of my proposal [here](https://docs.google.com/document/d/1BRJdEXm6vpUZWWvg7frCsz-CXPjCz6Mdy_DNDxjbmJE/edit#). Please have a look and revert me back with your feedback and valuable comments to make it better. I would like to remind you and everyone reading this that without doing the starter tasks, we will not consider your application at all, no matter how strong it is; > [@README: Starter tasks for LibreHealth Toolkit & Librehealth Radiology](https://forums.librehealth.io/t/readme-starter-tasks-for-librehealth-toolkit-librehealth-radiology/3016/): > > We have two issues that can help you showcase your skills, and help us evaluate if, along with your GSoC application, whether you will be suitable to work on the LibreHealth Toolkit and LibreHealth Radiology for GSoC 2021: [GSoC Intro: 3-tier architecture for FHIR Resource in lh-toolkit module (#40) · Issues · LibreHealth / LibreHealth Toolkit / lh-toolkit · GitLab](https://gitlab.com/librehealth/toolkit/lh-toolkit/issues/40)[GSoC Intro: Write a webcomponent with an existing REST resource (#41) · Issues · LibreHealth / LibreHealth Toolkit / lh-toolkit · G…](https://gitlab.com/librehealth/toolkit/lh-toolkit/issues/41) ---
**Author:** ![mshubhankar](https://forums.librehealth.io/user_avatar/forums.librehealth.io/mshubhankar/32/982_2.png) [@mshubhankar](https://forums.librehealth.io/u/mshubhankar)\ **Post date:** [March 31, 2019, 8:24pm UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/28 "2019-03-31T20:24:47Z")
I am sorry I missed the Intro task. I have now completed the Intro task 1 and my code can be found [here](https://github.com/mshubhankar/openmrs-module-messageheader). I hope you can now review my proposal. ---
**Author:** ![r0bby](https://forums.librehealth.io/user_avatar/forums.librehealth.io/r0bby/32/1781_2.png) [@r0bby](https://forums.librehealth.io/u/r0bby)\ **Post date:** [March 31, 2019, 11:14pm UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/29 "2019-03-31T23:14:47Z")
Link to it in your proposal. ---
**Author:** ![mshubhankar](https://forums.librehealth.io/user_avatar/forums.librehealth.io/mshubhankar/32/982_2.png) [@mshubhankar](https://forums.librehealth.io/u/mshubhankar)\ **Post date:** [March 31, 2019, 11:29pm UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/30 "2019-03-31T23:29:50Z")
I have added it under the “Contributions and work done” section of my proposal 😃 ---
**Author:** ![prashadi](https://forums.librehealth.io/user_avatar/forums.librehealth.io/prashadi/32/460_2.png) [@prashadi](https://forums.librehealth.io/u/prashadi)\ **Post date:** [April 1, 2019, 2:40am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/31 "2019-04-01T02:40:39Z")
@mshubhankar Sorry for the delay. I’ll have a look at the proposal and get back to you. Was away just over a week. ---
**Author:** ![judywawira](https://forums.librehealth.io/user_avatar/forums.librehealth.io/judywawira/32/264_2.png) [@judywawira](https://forums.librehealth.io/u/judywawira)\ **Post date:** [April 2, 2019, 3:56am UTC](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845/32 "2019-04-02T03:56:10Z")
@mshubhankar great start on your proposal I don’t see you recognizing the work @prashadi worked on last year in GSOC , and how your work will improve that This year the outcome of this project is to make it usable - esp to non technical users , so expect a UI improvement too [Next page](https://forums.librehealth.io/t/project-improving-fhir-analytics-using-apache-spark-and-cassandra/2845.md?page=2)