Project: FHIR Analytics Using Apache Spark and Cassandra

toolkit
gsoc2018-project
fhir
gsoc2018

(Yash D. Saraf) #23

@prashadi For executing CRUD operations will you be using spring data repositories or making REST API calls?
If it’s the former, you can use the PatientRepository directly, if it’s the latter then only READ and DELETE operations are functional right now.


(Prashadi) #24

For Analytics, I need to read the data directly from the Cassandra. Because I’ll be using Spark Cassandra connector to access data. I’m currently working on UI and resource analytic. You can work on adding few more resources.


(Prashadi) #25

@sunbiz @namratanehete @judywawira Please find my current progress in https://medium.com/@prkpbandara/gsoc-librehealth-librehealth-fhir-analytic-capabilities-c35f57e36a29. Since functionalities of UIs are completed, my next target is to integrate this module with @yashdsaraf project. Since he almost completed with patient resource, I believe I can test the module using patient resource. Let me know if you have any suggestions or improvements after going through the blog posts.


(Prashadi) #26

Interesting article which use amazon Alexa with FHIR https://medium.com/@alastairallen/evolve-and-alexa-on-fhir-30041e49bc51


(Namratanehete) #27

Hey @prashadi, I am getting following error after the latest update of the code. Please let me how to fix this error.

https://pastebin.com/800YQhE5


(Prashadi) #28

@namratanehete it’s strange as it’s working fine in my machine. Are you trying this war file with apache-tomcat-8.5.31 and jdk1.8.0_111?


(Prashadi) #29

@namratanehete I added another update as well with spring auto configuration. Please take latest update of the code and try it.


(Namratanehete) #30

Ok let me take update and get back to you. Thank you


(Prashadi) #31

Hi All,

This is to give an update about my current task which is integrating analytic component with spring data module. During past week, I have fully worked on integrating the spring data module and the analytic module. After several complex dependency issues, I managed to sort out them one by one. Since apply all of my changes to spring data module is adding too much changes, I have integrate spring data with the analytic module which we can later rename and push to spring data module after functionality is working fine.

I have hit a issue with using RouterFunction and RestController together https://jira.spring.io/browse/SPR-15405. The issue mentioned that both can be used together. I can see that all the request mappers are getting registered. But I can’t access spring data REST APIs with the given URL. @yashdsaraf would be able to have a quick look at why the Spring Data REST APIs aren’t working? Integration can be found in https://gitlab.com/kavindya89/librehealth-fhir-analytics/tree/integration.

@sunbiz @namratanehete @judywawira I’ll be continue to check this issue in my side as well. This is the only blocker that I encounter during the integration of two modules.


(Yash D. Saraf) #32

@prashadi I have checked out the integration branch, I can see the routerfunctions are loaded but are not accessible. However when I switched to using the rest controller instead it seems to be working. Although I can’t test it since I’m getting this error while sample data is being loaded

Exception in thread "restartedMain" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: java.lang.NullPointerException
	at org.librehealth.fhir.analytics.cassandra.CassandraDataServiceImpl.insertDemoData(CassandraDataServiceImpl.java:108)
	at org.librehealth.fhir.LibreHealthFHIRAnalyticsApplication.init(LibreHealthFHIRAnalyticsApplication.java:78)
	at org.librehealth.fhir.LibreHealthFHIRAnalyticsApplication.main(LibreHealthFHIRAnalyticsApplication.java:64)
	... 5 more

It would be great if you know how to fix it, as soon as I’m able to fix it myself I’ll make a pull request with all RouterFunctions switched to RestControllers.

Update

@prashadi The error I mentioned was just a file separator not being parsed correctly in windows for the DATA_PATH constant. The data is being loaded correctly now and I’m able to access both our namespaces’ data using cqlsh, but the REST query always returns empty. it doesn’t return a code 404 though, just an empty JSON object with code 200.


(Prashadi) #33

That’s great to hear. Any possibility on debugging the REST API? Let me know how I can help on this matter.


(Yash D. Saraf) #34

I haven’t been able to track down the cause of the issue yet, I’ll make a pull request with routerfunctions migrated to restcontrollers so you can debug the issue as well.


(Prashadi) #35

That’s great :). Thank you.


(Yash D. Saraf) #36

@prashadi I’ve created a merge request on gitlab.
Also if you are able to use my project as a single package (from what I’ve seen it’s platform), you might want to add my project as a git submodule, see - git submodules. That might serve you better than merging my repository’s changes manually. Just my two cents :slight_smile: .


(Prashadi) #37

Thank you. Sure I will do that.


(Prashadi) #38

@yashdsaraf When I debug get patient by ID method, I got following stackstrace.

reactor.core.publisher.Flux.next(Flux.java:5173) > org.springframework.data.cassandra.core.ReactiveCassandraTemplate.selectOne(ReactiveCassandraTemplate.java:252)

org.springframework.data.cassandra.core.ReactiveCassandraTemplate.selectOneById(ReactiveCassandraTemplate.java:364) org.springframework.data.cassandra.repository.support.SimpleReactiveCassandraRepository.findById(SimpleReactiveCassandraRepository.java:181) org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:377) org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:200) org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:629) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:593) org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:578) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:59) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) org.springframework.data.repository.core.support.SurroundingTransactionDetectorMethodInterceptor.invoke(SurroundingTransactionDetectorMethodInterceptor.java:61) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) org.librehealth.fhir.platform.controller.PatientController.getById(PatientController.java:133) org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) javax.servlet.http.HttpServlet.service(HttpServlet.java:635) org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) javax.servlet.http.HttpServlet.service(HttpServlet.java:742) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)

Do you have any idea about this? Can this be a dependency issue?


(Yash D. Saraf) #39

I’m pretty sure it’s not a dependency issue, what was the thrown exception?
Sorry for the late reply I was trying to get Observation resource to work in my project, I’ll look into debugging this now.


(Prashadi) #40

@yashdsaraf This the only available trace which I dig in and obtain via the debugger. Doing several things to get this working at the moment.


(Prashadi) #41

@yashdsaraf I can confirm that data is fetching correctly. I added following code to check whether data is fetching correctly in search method which gives the results. Only issue is that response isn’t coming.

List<CPatient> patients = patientRepository.findAll().collectList().block();


(Prashadi) #42

I change the search method as follow and then it’s started to work fine. I hope we can use this method.

 @GetMapping
 public ResponseEntity << ? > search(@RequestParam Map < String, String > params) {

  if (params.size() == 0) {
   // Return all entities if no search parameters specified
   List < CPatient > data = patientRepository.findAll().collectList().block();
   return new ResponseEntity(data, new HttpHeaders(), HttpStatus.OK);
  }

  try {
   List < CriteriaDefinition > criteriaDefinitions = new ArrayList < > ();
   for (Map.Entry < String, String > entry:
    params.entrySet()) {
    if (entry.getValue() == null) {
     throw new IllegalStateException("Search parameter value cannot be null");
    }
    criteriaDefinitions
     .add(getCriteriaDefinition(entry.getKey(), entry.getValue()));
   }

   reactiveCassandraOperations.select(Query.empty(), CPatient.class).subscribe(System.out::println);

   Flux < CPatient > patients = reactiveCassandraOperations
    .select(Query.query(criteriaDefinitions).withAllowFiltering(), CPatient.class);
   return new ResponseEntity(patients.collectList().block(), new HttpHeaders(), HttpStatus.OK);
  } catch (IllegalStateException ex) {
   return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(BodyInserters.fromObject(ex.getMessage()));
  }
 }