navarre.aiClarisTalk AI · the FileMaker AI podcast
Dr. Michael Schuckers on Machine Learning, CoreML, AI, Biometric scanning ...and Fantasy Football
Chris and Matt welcome Dr. Michael Schuckers to talk about Machine Learning. Two key aspects of ML are Features and Target: It's more about prediction based on data and different than the LLMs we have been discussing. LISTEN TO THE END FOR SOME CRAZY AI MUSIC Show notes: Papers by Dr. Schuckers: https://myslu.stlawu.edu/~msch/papers/BIOSIG2023_EquityTesting_MSchuckers.pdf https://www.biometricupda
Apple PodcastsSpotifyYouTubeAll episodes
Transcript
Transcribed by machine (whisper, large model), 10,613 words. Names and product terms are sometimes misheard; the audio is the record.
Claris Talk AI. Hey, Chris, it's another episode. Oh, my God, I'm so excited. We have a special guest. Yes, we do. Dr. Shuckers. Welcome, sir. Hey, thank you. How do you know this guy? Oh, you're talking, was that for me? It's really for either of you, because I know you guys know each other. I, you know, shadowy rooms in large conferences in the back, you know. Doing our snarky little comments about the lack of analytics discussion. Yeah. Yeah, I'm proud to say that Dr. Michael Shuckers, who he will not allow us to call him doctor anymore after this, but he and I have been collaborating on machine learning for, Michael, it's been like, what, 13, 14 years now.
We actually co-authored a paper as a little bit of a tease into this conversation back in 2013. And we disrupted an aspect of the sports community as well together. So how's that for a little tease? Actually, Matt, you've got another tease for a little bit later on. I have another tease. I'm going to show, I'll show a little clip for the YouTube people and then say what it is for the podcast people. But basically, this is going to be a very, very big reason to stick around to the end of this episode, because there's a site that Chris challenged me to look at called, I guess, UDEO. And it's AI music creation. And I will leave it at that because there's going to be two songs we'll play at the end that will be surprising.
And actually, for those podcast listeners, you may have heard a different intro song on this episode. And that was also part of that. So a little teaser for the end there. And in the meantime, the focus of today's conversation is going to be about machine learning and also contrasting machine learning and AI, because they are not the same thing. And I think it's important for the folks who listen to us talk about AI to know that machine learning is also a tool available to them. Before we get to that, I kind of want to know a little bit more about Dr. Shuckers. Your background, like what you do, because you're not in the FileMaker community. You're like, you know.
Yeah, no. So my day job is as a statistics and data science professor at St. Lawrence University. So it's a small, maybe prestigious hockey powerhouse in northern New York. Yeah. Women's hockey powerhouse. Yeah, for sure. And I guess men's lacrosse this year at a lower level. So, you know, really niche sports there. A little bit. But, yeah, I mean, my background is sort of in math and stats and have gotten more and more into machine learning. And Chris mentioned some projects we've worked on and, you know, been doing that over the last dozen or so years. Various applications. A lot of what I've been doing has been in sports and some other applications with fingerprints and iris scans.
And that's sort of biometrics world, too. So, yeah. Actually, I would say from a sports. So one of the annual events that Michael and I used to attend was the MIT Sloan Sports Analytics Conference, which I think we attended probably a handful of those together until finally we realized that it wasn't really an analytics conference anymore. It was just a sports conference from ESPN. And actually, most of the panels were just up there talking about how. They don't use analytics. However, Dr. Shuckers was on those panels talking about how he does mostly in the focus of hockey. Is that fair to say, Michael? Yeah, I mean, so I'm 15 miles from the Canadian border here.
And so the thing that gets our students most excited is ice hockey. So I've done a lot of work in that area. Including with the NHL, if I remember correctly. Yeah, so I've consulted with an NHL team, helped them with some machine learning stuff. To choose better draft picks. And and by the way, I would say, you know, from an academic standpoint, your influence is vast. I happen to know of various different former students who are peppered throughout the professional sports analytics community at NFL teams. And I would assume within the NHL space as well. Yeah. So I've got four former students who are working for NHL teams and two students who. Who are working for NFL teams.
And I could say some of those students that I mentioned for the NHL are actually working for the league office. So in the league offices, helping them do analysis. And so. So, Matt, Michael and I met because, well, I was having I was trying to do something in FileMaker that I couldn't do. And nothing FileMaker can't do. Well. I think that's an important point for today's discussion. Yeah. There are very few things FileMaker can't do. And what I was trying to do. Yeah. I can't bake bread. I cannot fix your back. Like, for example. Right. Those things that cannot do as much as you've got. You carry body soil, apparently. It can't carry body soil. Optimist is coming for that task, though, certainly.
But what I was trying to do many moons ago. This is probably like 20 2007 or something like that. Yeah. I had a FileMaker database where very important database. I was trying to track NFL player statistics so that I could optimize my predictions for their performance on Sundays, mostly just to try to beat my friends and family at fantasy football. So, again, grown man, the Las Vegas thing and the millions of dollars of money that you lost. And well, that was before Michael's influence. So I was actually. Trying to kind of put it into FileMaker speak. I had a like an average of what a player would do at his position. Right. And then I had these defenses and the average of what how they would perform against a position.
So just imagine like a quarterback every week. He goes out and he does these averages. Right. Right. Now, my like aha moment that, by the way, ended up being actually pretty profound because I later went on to, you know, work with the NFL and and the machine learning model. So that Michael and I worked on together were were used in those endeavors. But the aha moment was, well, what good is an average if all I really care about is who is that quarterback, let's say, facing this weekend? Are they a really good defense against the quarterback position or are they a really bad defense or where do they fall in that continuum in the middle? Yeah. And it may not be like the specific players that the quarterback is against, but other players with similar stats, that quarterback kind of kind of a thing.
There you go. Right. And to kind of. To kind of incorporate that together to find how did players with similar capabilities face against this defense? And it kind of starts to fold back on itself a bit. Right. So in FileMaker, I had like a list. And in my mind, I was trying to chase like this idea of a weighted average. And I didn't realize that averages really aren't very useful. And waiting meant something totally different than I thought. So Michael and I, I believe we actually met the first time in person, certainly at one of these MIT Sloan Sports Analytics Conference. So Michael, when I expressed to you what I was trying to do in FileMaker, which, of course, you said FileMaker.
Oh, yes. I've heard of FileMaker. From that movie that I saw. Problemista. Right. Yeah. So, Michael, I came to you with that problem and immediately you identified that I probably wouldn't be able to achieve this without machine learning. Why was that? Well, my assumption is that that was just not something that FileMaker had the built-in tools for. So, I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were.
And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And I was trying to figure out what the tools were. And adjust, right? What is the factors we can attribute to each? So what can I say is really the impact of a quarterback? What is the impact of the defense? What's the impact of home and away? And do that simultaneously.
And so mathematics has some tools, generally fall under the heading of regression, as Chris said, for doing that. Is that like a straight up calculation kind of a thing? Or is it running a bunch of different tests and averaging the results of those tests? Or how does that work? It's a pretty straightforward calculation if you've got the computational tools to do it. And it can generally be done in a sort of a single command sort of thing. Again, you got to know the right commands. I'm sure that's true in FileMaker also. Yeah, but I don't think those tools are built in. There's probably like what some custom functions or some other ways, but definitely not as powerful as machine learning.
So I'll shut up and let you continue. Well, I mean, we do just for whatever it's worth, have the ability to run core ML models inside FileMaker containers. That's not what we were using in this case, because, well, these efforts predated the core ML, but core ML can only run on a Mac operating system. And so if you want to go with something a little bit more versatile, you can create your own models and potentially make them as API sources. So you can send the variables over and get those back. Actually, I shouldn't say variables. I should say features probably is the big takeaway. There's really two pieces of vocabulary that Michael taught me, and that was features and targets.
And Michael, I'll let you talk about what those mean in the context of what you just shared with us. Sure. The target is sort of your outcome and the features tend to think of as the input. And so in machine learning, they're called targets and features. In the statistical community, they're words. They are response and covariance. Psychology calls them dependent and independent variables. And I guess that's most of social sciences. There's some various terms there that are pretty much interchangeable, but it's helpful to be able to translate across those different areas. But in terms of the machine learning, the AI community, it really is the target, which is that thing that I'm trying to predict, and the features, which are the things that I use to make those predictions.
So let's say my goal was, I wanted to be able to come up with what the right asking price of a home would be, right? I think that's a good kind of down to earth example. So you're going to sell a home, you want to do the right asking price. So let's say the data I have is a bunch of homes that have already sold. And I've got different, let's say they're in a FileMaker database. I've got a field for the zip code where that home exists, the number of bedrooms, the square footage, like all the different stuff, right? Now, I also have a field in there for what that home sold for, right? So in this case, we have a bunch of historical data. So Michael is, so the features of the home, the bedrooms and the zip code and everything, those are the features that we're talking about.
So the target in that case would be what, in that training data set? - Yeah, the target would be the selling price. - So in our historical data, the amount or what the home actually sold for. So I think this is key. We need to have historical data to create a machine learning model. I can't just say, let's create a machine learning model that tells us what the right asking price of a home would be unless I have, you know, a bunch. And by the way, Michael, where do you stand on small amounts of data for training versus large amounts of data? Settle it now, settle it now. - The more data, the better generally, right? As long as it's, you know, relevant to the problem at hand, right?
I'm not going to take data on home prices and sales in Las Vegas if I'm trying to buy a house in Charlotte or in Boston. - Right, right, right. And also like age of the data, you don't want data from the seventies or whatever. - Probably not, yeah. - Although like having that long, you know, the spike, and I guess another thing that occurs to me is you could also have the algorithm test each record against itself. So you can say this house actually sold for 620,000. If I guess what that house would have been sold for using the same data set, what would the guess be? And then you can compare those to the actual selling prices, that kind of thing. - Yep, and that's actually one of the ways that we sort of evaluate how well a model's doing, right?
We look at what we actually have historically versus what the model predicts, and we look at those differences. And of course, obviously the smaller we can get those differences, the better our model's doing. - Right, and actually that's one of the best parts of machine learning. When we were working for multiple years on the fantasy-o-matic machine learning model, or actually kind of a series of machine learning models in fairness, one for each position, right? Because the features are different from position to position. But when we would do that, we would end the season and we would talk over the summer about, hey, I wonder if we introduced passing targets or the age of the player or all these different features into our conversation.
And so what Michael would do is he would go take that same data that we originally trained the original models on, and he would add those. I'll let you talk, Michael, about what you were doing. And the benefit was, of course, we could then go test it on that historical data to see if we got better predictions. So Michael, what were you doing with all those variables when you were introducing new features? - You know, Chris, I don't remember, no. You know, what we would do is we would consider, you know, consider new features each summer, we would sit down and brainstorm and think about what are some, maybe some things that might improve, you know, the performance of our model.
And we would think about adding new features. Chris mentioned some of those, you know, player targets, player age, you know, age of the quarterback for a wide receiver, you know, some of those sort of esoteric things that maybe aren't really esoteric, that might really actually be helpful and, you know, the beauty of the machine learning is that we can go through and we can evaluate how well we end up doing with our prediction. - That's what was cool. - That's what we were doing. - We didn't have to wait till the next season and just cross our fingers and hope that those adjustments went. Michael would go run it on like the past 12 years worth of NFL data and see if our predictions got closer to what the actuals were.
And that's really the magic of machine learning is that you can do that at any stage. - Yeah, this is Moneyball plus AI though, huh? - Yeah, yeah, no, we're a big Moneyball fans. Yeah, well, I mean, we introduced AI into that equation later on as just a way for people to interact with the data in a more conversational manner, but- - I should have said machine learning. - Yeah, machine learning. - But I'm trying to remember if machine learning was even used in the movie Moneyball or if that was before that could have been done. I don't know. - Oh, it would have had to be. I mean, the models that they speak of would have to be machine learning and that, I think that's important to machine, although AI technically, goes back to, you know, the sixties.
The AI we're talking about these days is like language model AI. And that of course only goes back about five or six years, but machine learning goes back, what, even, you know, even for longer ago than the discussions on artificial intelligence, if I'm not mistaken, right, Michael? - Yeah, that's certainly true. And a lot of the things that we think of as machine learning weren't really called machine learning until about 10, 15 years ago. - Interesting. - So a lot of these techniques, these techniques are developed sort of independently and they come up separately with sort of under different headings, but, you know, machine learning sounds sexier than, you know, principal components analysis.
- Can you give me an example of something from a long time ago that we would now label machine learning, but wasn't at the time? - Sure. So the regression modeling that I was talking about, you know, that actually probably goes back 150 years, you know, in terms of, sort of, the basic technique. It gets some bumps computationally, you know, in the fifties and sixties, and then sort of hyper sped up around 2000 in terms of what we can do there. But, you know, the basic general idea has been around a long time, but it is now part of sort of this broader umbrella of what we think of as techniques in machine learning. - I think, well, so the other part that I think is really interesting is most of the people who listen to, you know, listen to us are in the business space, right?
And, you know, while this discussion, well, actually, so we were talking maybe about three or four years on just fantasy sports stuff and going to these sports conferences and just kind of like, you know, thinking about machine learning from that respect. And then it occurred to me, wait a minute, I work with businesses with data all the time. And our conversations then changed to actually using machine learning, this was well before language modeling AI, even. - Mm-hmm. - To actually bring that into the business forum. So I'm proud to say that, you know, as of today, Michael collaborates with us on a regular basis with our business customers. So maybe we can bring in, so actually, and just yesterday, we just did a AI session on machine learning and AI for one of our customers.
I'll humble brag and say that was 3M. They were nice enough to have Michael and I come in and talk about the very thing. Like, what is the difference between machine learning and AI? Michael, I'm gonna let you address, you know, this, you know, that difference from your perspective first. - So from my perspective, machine learning is a lot of the things we're talking about with features and with targets, right? And so building these models and connecting those things. And AI is really more of the, I guess, what I think of as generative AI, right? Which is the large language model things and the sort of moving a little bit, beyond what is your historical data to bring in some of these language models and make it something that is a little bit more interactive and thus generative.
- Yeah, I would say that, well, first of all, yes, I would say that, you know, in the modern discussion, AI is language model integrations. There are other components of AI, but when people are talking about it today, we're talking about that conversational aspect of AI. We're talking about some of the superpowers it has from a language perspective, but it does not, just to be clear, on its own, language models do not have the ability to make predictions. They don't have the ability to run regression, for example, like we've been talking about here. We need machine learning models to do that. Now, for those of you that use ChatGPT, or for example, like our iSolutions, our intuition execution platform is a Python execution environment that, you know, combines orchestration by language models together with tons of different Python packages that can do machine learning, or even spin up a model on a fly for that matter, or run the models that are actually created.
So working in conjunction with language models and machine learning together creates some like amazing combinations of technology, and those are very much available today. Michael, you just did a really interesting paper, if you don't mind talking about it. I just think it's fascinating. First of all, what was the, I mean, you can talk about the iris scanning face recognition one, right? Just before we, okay, good. So we don't need to edit this part out. Talk a little bit about who commissioned that and why, and what your findings were, 'cause I think it's fascinating. - I can't talk about who commissioned it, but I can say it's an arm of the U.S. federal government.
How about that? We'll start with that part. - All right, that's fair. We don't need you disappearing. - Yeah, yeah, I'm gonna give, so, for quite a while, actually before meeting Chris and before getting really heavily involved in sort of sports analytics and sports machine learning, did a bunch of work sort of in a post 9/11 world on how well iris scanning, facial recognition systems, fingerprints work. And those of us old enough to remember 20 years ago, right? You know, that fingerprint scanner on your laptop, was not as good back then as it is today. And so I developed some of the sort of original statistical methods for evaluating how well that performance goes.
And, you know, I'll be the geeky stat professor for one second, but there's this weird stat correlation structure that happens because you're comparing two images that are possibly from the same person or from a different person that create some weird variance there that, you know, we've got to account for when we do, you know, the sorts of modeling stuff that we do. So all that's background to, you know, the last couple of years we've seen a lot of press and a lot of publicity on issues of, in particular with face recognition, differences across demographics, differences across genders, differences across race, ethnicity, sort of factors there in terms of, you know, the performance.
And certainly, you know, there have been some tests where we're finding differences across race and ethnicity for different devices or different algorithms that are doing these sort of matching. And so, you know, I pulled on my old biometrics hat and I got involved in this test. And what we're doing is we've got five different devices. These are all facial recognition systems. And we're basically doing the statistics to say, well, what's sort of the margin of error that can happen around those error rates to say, yeah, all of the demographics are equal, right? You know, we've got a large test. I think we've got order of magnitude about 4,000 subjects, you know, as diverse as we could get it.
And, you know, looking at differences among the genders, differences among the race, we've got education levels, we've got incomes levels there. And looking between these different groups, and saying, you know, what is a real difference here and what is just kind of noise in the system? And so that paper for that methodology has come out. I think in the next six months, we'll actually be able to publish and I can tell you who published it. But, you know, it's one of those things that has to go through 17 layers of the federal government to get approval to talk about something like this. - Especially in an election year. - Right, yeah. Yeah, I think we are getting close to wrapping up their particular study.
So we started having this conversation 16 months ago. Over the last 12 months, I sort of developed and published the methodology. And then we're gonna get the results for the study coming out, hopefully like Q3, Q4 of 2024. - And in short, there's a ruler suggestion in there? - Yeah, I mean, what it looks like is, you know, you have a sort of average error rate for everybody, and then we create this sort of, you know, plus or minus like you would get from polling, right? And so it's gonna be, you know, maybe the device has a 6% error rate and it's gonna be, yeah, everybody should be plus or minus 2%. And if you fall outside that, we're gonna say, you know, that that demographic subgroup is different, right?
Whether it performs better or worse. - And this is a way that you can calibrate fairness and into face recognition technologies, which is really the goal. - So we're trying with this sort of operational definition of fairness to calibrate that, yep. - What is the actual percentage of reliability and facial recognition with today's technology? - So it very much depends on the device, right? Your phone, which is really just doing one-to-one, just you to you. There shouldn't be much changes over time. That's gonna be really high, right? You know, the error rate there is probably less than 1%. You know, you get your hair cut, you know, you throw on sunglasses or you get different frames on your glasses or, you know, you're gonna, you know.
- You're waking up in bed and only half of your face is showing and you're trying to unlock it so you can play. - I feel like during, I feel like I've gained a lot of weight over the holidays before and my face ID stopped recognizing me, so. - I think I got outside that 1% on a couple of Christmases. - Yeah, so those sorts of things are gonna be dynamite. - Right, but then like, but like telling a person from a pool of a million. - Yeah, much higher error rates, right? And you know, that's just gonna, you know, the false accepts and the false rejects are gonna go way up the larger you get for pools of things like that. The particular thing that we were working on, for the federal government is we have them take a selfie and have them take a picture of their driver's license and trying to match that.
And, you know, as you can imagine, right, my driver's license picture is a little out of date. And so, you know, the error rates are certainly more than, I don't know, one or 2% for those kinds of circumstances. And admittedly, those are hard problems, right? The image quality from a selfie can vary a lot. - Yeah. - We see that across age, age is a really big factor in quality of selfie taking, it turns out. - Yeah, it makes sense. Plus driver's license pictures are gonna vary a lot. I mean, as I think back over the last year, I think there's been a number of times where I've had to do exactly what you're talking about. Like take a selfie and take a picture of something.
Like for example, if you use Wise or to have money in different currencies, which you absolutely need to do if you, if you spend a lot much time in Europe or live there. Anyway, yeah. - So speaking of images changing over time, can I just, can I just hear something? - Yes, yes, yes. - So the folks in the, so for, this is for our YouTube friends. - Chris Anthony. - Whoops, oh, sorry. This is actually a piece that's available on clairs.com that they did about the work that, we'll link to this in the show notes, but this is a FileMaker application where we integrated in the machine learning model that Michael and I chose. This is a video of me as a young man, but this was, I look pretty much, I look the same.
I got the same shirt on here, right? Just for those folks at home. But if I scrub forward here, oh, look at that. - I'm not seeing anything. I'm just seeing a frozen screen of a computer. - I'm just seeing your laptop screen. - Oh, I see what happened. Okay, gotcha. So yeah, let me just do the full screen here for you guys. There you go. Yeah, look at this. So anyways, there's Michael looking young. Oh, actually a little, there we go. See, look how much younger you look. - Oh, goodness, yeah. - So the reason I wanted to share this is 'cause Michael and I are very proud of this. This is just for our YouTube folks. Matt, you can describe what you're seeing on screen.
- Someone throwing a football really poorly. - Hey, that was a- - That was a pretty nice spiral. That was a nice spiral. - It could be better, but former Shorewood High School, Greyhound quarterback, Chris Ippolite throwing- - That look of concentration on your face, excellent. - But look at that pad. - Also an excellent catch. - Michael ran across, yeah, I would say that that's the important part is the catch. So with that is a little bit of fun. So Michael, I think, you know, yesterday in our presentation for our customer, we were asked, when can you identify a good use case for machine learning? And I think that's probably a good thing to talk about, you know, in the spirit of trying to encourage our Claris folks to explore machine learning themselves too.
What do you think there, Michael? What's a good way to frame up when, and I have an anecdote about a conversation you and I shared too, after that to kind of help frame that up too. - Sure, yeah, no, it's a great question, Chris. You know, I think when we've got sort of a well-defined target and we've got a well-defined set of features that we want to use to make those sorts of predictions and that we have, you know, good amounts of relevant data, right? You know, we were talking about relevance of data a little bit earlier, right? And so, you know, we might be able to take, you know, I'm thinking of like, you know, if I'm in Washington, D.C., I might have a slightly different model than I have in the suburbs, but I might be able to use some of the data from the suburbs to help inform that sort of, you know, model within the city, but I'm not so sure I want to extend that out to Philadelphia or Boston, right?
- Right. - And so, you know, having a good amount of relevant data, again, you know, it can be hard to judge how well some features are going to do. You know, Chris, we have done a bunch of things where we thought we had all the right features and we go in and, you know, the performance of that model is just, you know, garbage, to be perfectly honest. - Those are potential outcomes, yeah. - You know, and the other thing is in machine learning, there are different sets of models that require, you know, different sorts of inputs and different ways to sort of attack these problems. And, you know, each of them requires a different sort of magnitude of data to help inform, you know, that model and how well it predicts.
- You know, just last week we had a couple, or on our last episode, we had a couple of special guests from the FileMaker Claris community. - Mm-hmm. - And the week prior to that episode, those same guests and myself were on a webinar for Claris. And one of the examples that I showed in that webinar, and people can Google that, Claris AI webinar, was the project we did for the company in Chicago, Michael, that had the estimating application. And so Michael created the machine learning model there. And as an example, what we had was an estimating application in FileMaker that, that also then you could convert your estimate into a job, but they would put in the job codes that they thought were gonna be necessary to complete the task.
And then after the work was actually performed, they would have the actual job codes that were required. So in that case, Michael, we gave you, I don't know, it was like a couple thousand or, I don't know, I think it was like a thousand or something of the jobs that were, the estimates that were actually converted into jobs. And so in that case, how were you able to model that data? And why did that model work because, you know, like, why was it that we were able to do it in that case, with the data set? - Yeah, I mean, this is one case where, you know, I think the first thing that we tried, which was doing, you know, looking at sort of what were the keywords that were in the description for the particular job that they were trying to estimate the costs on, and using those keywords to try and do some prediction.
You know, we tried that and it didn't work. And, you know, we were off often by quite a bit. And, you know, that happens. This is not a linear process, right? Sometimes gotta try and fail and try again. And, you know, then we tried and that first attempt was using something that was akin to the regression sort of modeling that we were talking about earlier. - Yep. - And, you know, after trying to beat that one to death, then trying to get the, you know, extract the, best model we could out of it, we weren't getting anywhere. And so we tried to go to a whole other class of models. And we went to a class of models called nearest neighbors. And they are, what we're doing is we're trying to look at the description we have for the job and then find other jobs they've done that were as similar to that as possible.
And that turned out to be a really good way of doing it. And that's our performance. I think like Chris, we were able to go from like 12% to like 80%. - Yeah, that's why I didn't want to say that. - In terms of our predictive ability by switching those models, so. - Yeah, I mean, you know, you can imagine that estimating in any company anywhere is very low probability of a hit rate. I mean, it depends on how sophisticated the actual job tasks are. In this case, they were down in the 12% of accuracy. And we were able to raise that up, you know, through the modeling process significantly into the high 70s, as I think most, any company that really looks at their estimating versus actual process could probably, you know, appreciate those gains.
- Yeah. - And that's just straight ROI. That's like literally money in the bank, right? On efficiencies. So that's why I think it's so critically important for people to be aware of machine learning as a tool to help with business. And the other factor is because, well, and I'll share this with you as well too. We're so close to these companies' data because that's literally why they hire us. So once, if you can just sort of, even if you don't understand the machine learning modeling process, if you can just kind of identify when a customer has a lot of really good data and it is potential for machine learning, then you're already like halfway there. And also the fact that we build custom models, I'll share this very brief anecdote.
One Monday morning, I was all shot out of a cannon. This was before COVID, Michael, but I called you up and I'm like, "Hey, I have this brilliant idea. It's called a project risk rating, right? So we have this project management system. And in regression, we're able to create these ratings. That's how the whole fantasy-o-matic thing was based on ratings. And by the way, just to kind of frame up ratings, Michael taught me this. I'm just gonna make him proud here. Hopefully, actually, I'm not BSing, but here's a good way. I actually just did this for someone else, a company named not to be mentioned, but they gave me, they didn't give me, I obtained a spreadsheet with a bunch of columns in it.
And it was essentially like a bunch of, companies that were performing at these different levels, like revenue generation or conversions or whatever, right? And I looked at it and I thought, well, I can't even make sense of what the company who's using this, who's in first place, like who's the top one of these companies that are listed, like their top sales reps or something, right? And because every column, if you sort by one column, it could be the most revenue generating. If you start by another one, it's the most closing of deals or whatever the deal is. So I instantly thought of this, same regression process and spun up a little notebook that I got very lucky that it worked, Michael, just so you know.
If this wasn't a Sunday morning, I would have bothered you with this. But anyways, I said, let me take each one of those columns and assign a weight to it. Meaning this column over here of revenue is more important than this one over here and so on and so on. And then ran that through a regression model and it spit out a rating, just a random number, you know, less than between negative one and one. But it can serve as a way to actually rank those. And so I was able to rank those reps, for example, and that spreadsheet. - Not really random number, but yeah. - Not random at all, actually, totally not random. - Exactly. - But you know, the point is, it doesn't like mean percentage of something or whatever.
- Right, right, it's just a relative score ranking for, yeah. - Yeah, and so in the spirit of that, this was years ago, I called, well, that was just last weekend, but then years ago in a similar fashion, I called up Michael and I said, I want a project risk rating. I wanna look at a project and tell me when there's risk involved and then rank them. And then that way I can go talk to the project managers on the highest potential risk ratings. And Michael said to me, and I thought this was very profound. He goes, well, he goes, Chris, you're a genius. I just wanted to let everyone know. He didn't say that at all. He said, okay, well, what data do you have? And I was like, oh man, I actually don't have any historical data because our system was not collecting the data or the features that I needed to give to Michael to, to be able to train on them.
So the reason I think that's an important story for FileMaker folks is, since it was a custom business application, we have control over our workflows and our data capturing mechanisms. We could then now change that process so that we were now capturing data to be able to model later down the road. So I think for our FileMaker folks, you're very close to the data. You're right there with this. You have the ability to change the data collection mechanism and influence that process greatly. And FileMaker, the Claris platform does have native support for machine learning models. And now in that case, it's core ML. But what I will tell you is like in the customer example, we just talked about with the estimating and actual, we took that model and then tested it.
And once the model worked, we then created APIs for the model. So we just put it up on a web server, super low cost. And now the FileMaker application sends API calls with those targets for a new job over to that API server. And it returns back with the responses from it. So we have-- - Yeah, so that makes it a cross platform. - 100% cross platform. And we have really good native tools for sending and receiving and parsing API calls within the Claris platform. So hopefully that inspires people to learn a little bit more about the impact of machine learning. - It's kind of funny that I never thought about like turning it into a tool to actually determine which jobs are more or less risky, which I think you have to be a pretty large consulting company to have enough jobs to do that.
But, you know, there's a number of those type companies. That's quite interesting. - Yeah, you gotta have the data and it has to be structured. I think the structure part of it, of the data, there is a concept, Michael, the difference between structured and unstructured machine learning for the sake of this conversation. - Yeah, that's a big one. Generally, if you can put it into a spreadsheet, then it's structured. And if you have issues where it's not, I mean, unstructured though, you know, some of the databases where you've got shared keys across different tables and those sorts of things, that gets into the murky land between structured and unstructured. But, you know, generally- - But you say- - If we get it in a spreadsheet, we think of it as structured.
- Now, it's not that structured is better and unstructured is worse. It's just different processes. But let's just put it this way. If I came to you, hey, new project, customer's really excited, their data is structured. Or if I came to you and said, hey, new customer, their data is unstructured, which one would you be most excited about? - Well, structured is just a little easier to work with and get a handle on, but you have just two different sets of tools, right? Two different toolboxes that you're gonna use, one for structured and one for unstructured. - And that's what I wanted to get out of you is that since we are also the ones who are building the systems for data collection and maintaining them, we have control over them.
We have control over the structure of the data at the same time too. So sometimes that could certainly be a benefit for us folks in the custom business application space. - Michael, what are we not asking you that we should be asking you? - Oh, that's a great question. - It wasn't even on the agenda. It's just something I wanna know. - Yeah, no, it's good. We've mostly been hitting on ML that I think of as supervised. So when we're talking, when we're talking about the I've got a target and I've got features, we think of that as supervised and it's really the target that defines whether or not we've got something that's supervised. There's a whole other area of machine learning that is mostly just digging into the features and looking for the patterns between the features.
- Not seeking a target, but looking for a pattern. - Yeah, not even a target. You're just looking to see what do I have here is my patterns. There's a lot of work there in clustering and saying these things are like these other things within my data. - Yeah, like I've noticed that quarterbacks when they play home versus away games have X difference in there. So it's not a specific target you're looking for, but it's something statistical and meaningful. How two statistics relate to each other. Is that a good synopsis? - Yeah, that's a good synopsis, right? If we were thinking about clustering quarterbacks, we might think of athletic versus non-athletic. And what are sort of the measures that we get that would put them into those two sort of clusters?
- Wait, non-athletic quarterbacks, is that a thing? - Yeah, Tom Brady, Matt Manning. - Post-Achilles, what's his name, Chris? What was his name again? - I don't know that guy's name. Not in my university anymore. Hashtag sports for all the big sports fans out there. So by the way, if we didn't, okay. So first of all, Michael, thank you so much for being here. I've been wanting to have a conversation with you to share with people for a long time. It is my true honor to be able to work and collaborate with you currently to this day. And our customers are all beneficiaries of that. It has been a fun, personal part of my own identity to work with you on the machine learning stuff on the fantasy-o-matic side.
And thank you so much for bringing machine learning into my life because everything that got me here, that got me curious about AI started with machine learning because mostly I started to do custom machine learning models with your guidance. And by the way, if we had recordings of the early conversations of Michael trying to explain these things to me, like it would make for a great pod, but probably be super embarrassing to me. But the point is I realized, hey, there are some models that need to be custom because of data. Then there's some that just could be generic ones, right? And I started to find those out in the cloud with like sentiment analysis or classical analysis.
- Mm-hmm. - Or classification or, you know, clustering. And then I realized, wait, there's, you know, language models out there that do all those things and more. And now, if I was being honest to wrap up the whole journey, not wrap it up, but now we're actually collaborating. As a matter of fact, that solution that we just talked about, the one that we showed in the webinar, was a combination of language models, parsing out plain English into inputs or features for Michael's machine learning models. So they can actually work side by side. - Mm-hmm. - And I also mentioned, you know, in these like execution environments, like our iSolutions Intuition environment or Code Interpreter, for example, at ChetGBT, those are language models orchestrating Python and machine learning models as well, too.
Now, I just said orchestrated. - Chris, if I can get a word in, and I've got some experience knowing how hard that can be. - So does Matt, by the way. - I would just thank you for all the fun conversations that we've had. It's been a pleasure, and really, I enjoy working on all the projects that we've done. So that's a blast. - Oh, thank you for that. Well, and I said a key word in there. - You did. - So now, Michael, you get to join us on some fun. I said a key word in there, orchestrate. So a fun fact about Matt. Matt, you want to tell everybody maybe a little something about your non-filemaker side that they might not be aware of? - I would say I'm like an amateur composer.
I write classical music. Choral music, a little bit of orchestral music. Yeah, that's kind of-- - And anybody who's listened to the podcast has heard a piece of your work, because the intro music of the podcast version is something that you wrote. I've always been fascinated by that aspect of you. And because every two weeks we talk about AI, one of the big things that came out a couple of weeks ago was AI music generation. We've seen the fantastic and amazing things with the Sora from OpenAI and the various different diffusion models for creating images and video. Now it was music's turn. So just like you challenged me in our last one to take your FileMaker proficiency exam, which I ended up having ChatGPT4 take.
Oh, actually-- - Oh, I have some updates on that, by the way. - Can we, yeah, should we talk about that? And then we'll just do some plugs, because I want to get into this fun music discussion to play us out, so to speak. So first of all, update on that, please. - A couple of my students have taken that test now and ChatGPT scores around the middle of like a pretty good FileMaker developer. - So-- - Surprisingly higher than I would have thought. - So don't tell any students when they score lower than ChatGPT though. Just tell them that they out, tell the ones that outscored ChatGPT, which would be a 78 in case anyone's listening. So Michael, we had a GPT4 take a FileMaker proficiency exam and it got a 78.
And throughout that discussion, we were also talking about how the latent intelligence in the GPT4 model was not 100%. But as Matt's indicating, 78 is pretty darn good though. Like it's operational. So one of the things as far as a plug is concerned, I just wanted to mention that somewhere in the show notes, we will, including a bunch of great information about Michael and the paper he talked about and some other neat things there too, like the video of us doing an incredible touchdown pass across the middle. We'll all be in the show notes, but also the FileMakerLearningCompanion.com initiative. I just scheduled a community meeting for Friday, May 31st at 8:00 AM Pacific, where we're inviting everybody to come in sort of a town hall fashion to start talking about what we can do there.
We'll have, you can sign up on FileMakerLearningCompanion.com in a Clare Studio form to become involved. Very excited about that. The other thing that I'll ask people to do, is in the show notes, there's also gonna be a link to vote for Chris's session. I did an AI session that I'm really proud of at Claris Engage. I've now doubled that in coolness. And, oh, thank you for the thumb bubble. - I just noticed that I was just like holding my arm and it's like it interprets that as a thumbs up. - That should be a back pain bubble, actually. - Yeah, I know. - So I'm proud to say there's this tech conference in Milwaukee where I'm from called Summerfest Tech. And I made it as a finalist for being a presenter and I need votes between now and May 24th.
So if you could click into the show notes, both on the podcast or the video and give me a vote, you can vote up to once a day, vote for my session, which is called AI Deployed Real World Examples. I really appreciate the support from our community. Now, enough about me, any plugs for you, Matt? - No plugs for me. I will have plugs for my upcoming classes, but that's not quite yet. - How about plugs for you, Michael? - Well, I'm gonna plug a project that I'm working on, which is the scorenetwork.org, where we're developing sports examples for statistics and data science education. And so we've got a network of educators, network of sports personalities, trying to build these sort of educational modules and trying to do so, yeah, there you go.
- I just went to it too. - Scorenetwork, and we got some funding from National Science Foundation. We're trying to build and enlarge the pool of students who would be interested in sports and data science, or in statistics and data science. And so, although some of those early ones are in some of the usual sports, we're trying to do a lot of things like triathlon and other sports that might not interest your typical sort of statistician. So. - Fantastic. - Yeah. - Awesome. So, then Matt, you don't have any plugs, but you've got some fun stuff to share. So let's get to the fun part of our discussion here. - So you gave me this assignment to check this site out and to see what I could do, right?
UDIO, beta. - So that's U-D-I-O dot-- - U-D-I-O dot com. I assume it's U-D-O as in studio without the "sta." - Mm. - 'Cause who needs the "sta?" - Yeah. - So you basically ask for a particular song. You can give it a prompt, basically. - And for those of you watching on the YouTube channel, you can see what the interface look like, that looks like, and the prompt entry. But really it's just entering a prompt at this point, right? - Come to think of it, I don't think I shared the sound. Let me reshare again. So yeah, you basically put in a prompt for whatever you want to do. And then, and then it creates a song. It takes a little longer. It takes a minute or something like that.
- So give us an example of a prompt. - And then it makes two versions of it. So like, I actually did one today that failed. I said, write a fugue classical in the style of Bach or harpsichord. - And then you have some other tags you can pick, like-- - Right, you can put in other things if you want. - Classical. - Yeah. So, and then it failed miserably on this one. All right. It was, yeah, A, it didn't write a fugue. B, it wasn't for harpsichord. And C, if you specify a particular artist, it says it can't, it can't simulate an artist without the artist's specific permission. And they can't get permission from Bach. - Can you say Bach-ish? - You actually wouldn't be able to say that.
You just have to say like Baroque or something like that. - Gotcha, okay. - And so then what happens is it gives you two, you know, two songs. I actually usually did them both 'cause it's so bad. But then I started some other ones, right? I started some like, like a classical tone poem. I don't know if you can hear a little bit of this. I don't think that's at all what I was asking for, but then there was a couple of them that I really, really like. I had to do like, you can, what do you call it? Remix it. So after you get one, you can ask for a remix, give it a couple more prompts, right? This is not a zero shot kind of a thing. So here's one, which I labeled intro et dominum, which is Latin for introduction and God, which means really nothing at all.
♪ Every song ♪ ♪ Of the blues we pretend ♪ ♪ Moved from high in the moon ♪ ♪ To heaven ♪ ♪ Oh ♪ ♪ Oh ♪ - So it's singing all of those parts. And I actually put in lyrics. I made in some Latin. It doesn't understand Latin, so I put in some other stuff. And you can't understand the words at all. But it, I can't anyway, listen to it. And I typed them in. - So, okay now, as a musician, as a composer, what's your initial instinct as to the quality of this? - I was shocked at, some of the things that it does well. So first of all, just out of thin air, it makes some of that music with some really interesting progressions. It's absolutely unique and creative. I don't know if I could use that for anything, but I mean, if you're writing, like doing a film and you just need some background music and you need to have a really specific mood, and this would be good for that.
- And then as a musician, well, great point. And then this is what I, I think hopefully, this is a useful framing. But as a musician, how much of being a musician do you think helps you personally create something better than if I sat down and did this? - I think a lot if you're gonna be sculpting it. Because like when I write a piece of music, I will have a really specific idea of what I want in terms of the sections, the arc of dynamics, the tonalities, when keys change, how long things go in a certain, like there's a lot of little details like that that for me, they're really clear in my head when I write it. If I can give those- - You can put that in, have you noticed you can put those in the prompts and it responds or not yet?
- No, not yet. I don't think so. I haven't actually really tried to do that. Like, because it would be really, really detailed, you know, like write a symphonic overture, which is 86 measures. The first two measures are an intro starting in B flat, minor, blah, blah, blah. I mean, it would just be super long. - Super long to try to describe something like that. - Well, but I think that's where this is going. I remember when I first started playing around with Midjourney a couple of years ago, you'd have to go on to Discord and like go into a chat and you could see what everybody else's prompts were for Midjourney, right? - Yeah. - And what I thought was so fascinating, it was people who like would specify a specific type of a camera, like circa certain, you know, like with certain filters or frames or whatever.
And I was like, oh wow. So people that know what they're doing can actually have better outputs than people who are just messing around with this. So it's, we're hopeful that that's, you know, the same thing in an environment like this. - Yeah. So, and I think in different genres, it's going to do very, very different. Okay, so one of the things that's really cool is you can actually put in your own lyrics. - So. - And so on a classical choral song, that's one thing, but if you do it with a rock song or something like that. - Oh, these are the lyrics right here? - These are the lyrics right here. So we're going to hear this in a little bit. So here's how I wrote these lyrics.
I went to ChatGPT and said, I want to write a song about FileMaker development. I want it to be blah, blah, blah. So I gave it some prompts and then went through, I didn't zero shot that either. And then I rewrote it because those outputs weren't very good and then had it fill some things, right? So it's basically like, you know, the way that I use ChatGPT a lot is cooperation, right? I get ideas from it and then I make a lot of other changes. - You're doing it right, yep. - Yeah, so for the people who are on audio, you'll step in the game with FileMaker Pro, jam your records in my table, debug and press go. Layout slick and my scripts on patrol, stick your record on my table, let's lose control, right?
- Wow, oh, we just got an explicit rating on the podcast for that last line there. - Oh, really? I wouldn't think so. Don't embarrass me with Claris megabits megabyte. - That was clever. I have to admit, that part, I listened to it. I listened to it a couple of times over. - Join your table if you're able, Chris. Code all night. I put your name in there. - That was adorable, thank you for that. - Okay, so how does it do for this, right? So with these silly things that I put in and I asked it to do like a rock song. This is probably just gonna be our go-outs or probably just gonna say goodbye and I'll just play this whole thing. - Yeah, we'll say goodbye and then we'll play it out and we'll let people listen to this on both YouTube or on podcasts.
So thank you, Michael. Really appreciate it. - Yes, thank you. - And we'll see everybody in a couple of weeks. Check us out on podcasts. Please look at the show notes. Bunch of good stuff in there. Check us out on podcasts and YouTube and Matt, let her rip. - And I'll put links to these songs too. ♪ Yo, step in the game ♪ ♪ With Farm Maker Pro ♪ ♪ Jam your records in my table ♪ ♪ D-bop and press go ♪ ♪ Layout slick ♪ ♪ And my scripts on patrol ♪ ♪ Stick your record in my table ♪ ♪ Let's lose control ♪ ♪ Don't embarrass me with Claris megabits megabyte ♪ ♪ Join your table if you're able, Chris ♪ ♪ Code all night ♪ ♪ Slide your panel ♪ ♪ In my portal ♪ ♪ Feel the flow ♪ ♪ Relationships linking back and forth ♪ ♪ We go higher up those scripts ♪ ♪ Set the show ♪ ♪ Validate my fields ♪ ♪ Data's starting to grow ♪ ♪ Yes ♪ ♪ Let's step in your heart ♪ ♪ Reliance ♪ ♪ Don't embarrass me with Claris megabits megabyte ♪ ♪ Join your table if you're able, Chris ♪ ♪ Code all night ♪ There's a panel in my portal, feel the flow, relationships linking like a force, we grow, grow, grow, grow.